Max level shown:
12025-04-30T06:42:48.940ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_crdb
22025-04-30T06:42:48.941ZDEBGtest_snapshot_unwind: Starting CRDB
32025-04-30T06:42:48.970ZINFOtest_snapshot_unwind: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpz1QErh
42025-04-30T06:42:48.970ZINFOtest_snapshot_unwind: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpz1QErh/data)
52025-04-30T06:42:49.001ZINFOtest_snapshot_unwind: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpz1QErh/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpz1QErh/listen-url
62025-04-30T06:42:49.002ZINFOtest_snapshot_unwind: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=4 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:42:50.661ZINFOtest_snapshot_unwind: cockroach pid: 27074
82025-04-30T06:42:50.661ZINFOtest_snapshot_unwind: cockroach listen URL: postgresql://root@[::1]:37592/omicron?sslmode=disable
92025-04-30T06:42:50.662ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_clickhouse
102025-04-30T06:42:50.662ZDEBGtest_snapshot_unwind: Starting Clickhouse
112025-04-30T06:42:51.193ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:42:51.193ZDEBGtest_snapshot_unwind: Starting Management Gateway
132025-04-30T06:42:51.319ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:42:51.319ZDEBGtest_snapshot_unwind: Starting Dendrite for switch0
152025-04-30T06:42:51.373ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:42:51.373ZDEBGtest_snapshot_unwind: Starting mgd for switch0
172025-04-30T06:42:51.489ZDEBGtest_snapshot_unwind: mgd port is 42076
182025-04-30T06:42:51.489ZDEBGtest_snapshot_unwind (init_with_steps): Running step record_switch_dns
192025-04-30T06:42:51.490ZDEBGtest_snapshot_unwind: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:42:51.490ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_internal_dns
212025-04-30T06:42:51.490ZINFOtest_snapshot_unwind (store): opening sled database path = /var/tmp/omicron_tmp/.tmpznoctz
222025-04-30T06:42:51.607ZINFOtest_snapshot_unwind (store): pruning trees for generations newer than 0
232025-04-30T06:42:51.607ZINFOtest_snapshot_unwind (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:42:51.607ZINFOtest_snapshot_unwind (dns): DNS server bound to address kind = dns local_address = [::1]:64164
252025-04-30T06:42:51.608ZDEBGtest_snapshot_unwind (http): registered endpoint kind = dns local_addr = [::1]:54858 method = GET path = /config versions = all
262025-04-30T06:42:51.608ZDEBGtest_snapshot_unwind (http): registered endpoint kind = dns local_addr = [::1]:54858 method = PUT path = /config versions = all
272025-04-30T06:42:51.608ZINFOtest_snapshot_unwind (http): listening kind = dns local_addr = [::1]:54858
282025-04-30T06:42:51.609ZDEBGtest_snapshot_unwind (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:54858
292025-04-30T06:42:51.609ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_external_dns
302025-04-30T06:42:51.609ZINFOtest_snapshot_unwind (store): opening sled database path = /var/tmp/omicron_tmp/.tmpbsCt6b
312025-04-30T06:42:51.627ZINFOtest_snapshot_unwind (store): pruning trees for generations newer than 0
322025-04-30T06:42:51.628ZINFOtest_snapshot_unwind (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:42:51.628ZINFOtest_snapshot_unwind (dns): DNS server bound to address kind = dns local_address = [::1]:45789
342025-04-30T06:42:51.628ZDEBGtest_snapshot_unwind (http): registered endpoint kind = dns local_addr = [::1]:40326 method = GET path = /config versions = all
352025-04-30T06:42:51.628ZDEBGtest_snapshot_unwind (http): registered endpoint kind = dns local_addr = [::1]:40326 method = PUT path = /config versions = all
362025-04-30T06:42:51.629ZINFOtest_snapshot_unwind (http): listening kind = dns local_addr = [::1]:40326
372025-04-30T06:42:51.629ZDEBGtest_snapshot_unwind (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:40326
382025-04-30T06:42:51.630ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_nexus_internal
392025-04-30T06:42:51.630ZDEBGtest_snapshot_unwind: Starting Nexus (internal API)
402025-04-30T06:42:51.630ZINFOtest_snapshot_unwind: setting up nexus server
412025-04-30T06:42:51.633ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Action
422025-04-30T06:42:51.633ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Database
462025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Fleet
482025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Inventory
492025-04-30T06:42:51.634ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:42:51.635ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Project
582025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Disk
592025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:42:51.636ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Instance
642025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = IpPool
652025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Vpc
672025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:42:51.637ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Image
752025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Certificate
802025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:42:51.638ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:42:51.639ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Rack
852025-04-30T06:42:51.639ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:42:51.639ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SshKey
872025-04-30T06:42:51.639ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Silo
882025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Sled
942025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:42:51.640ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = Service
1012025-04-30T06:42:51.641ZINFOtest_snapshot_unwind (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:42:51.642ZINFOtest_snapshot_unwind (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:42:51.675ZINFOtest_snapshot_unwind (ServerContext): Setting up resolver using DNS address: [::1]:64164
1042025-04-30T06:42:51.675ZINFOtest_snapshot_unwind (DnsResolver): new DNS resolver addresses = [[::1]:64164]
1052025-04-30T06:42:51.675ZINFOtest_snapshot_unwind (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:37592/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 37592,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:42:51.680ZDEBGtest_snapshot_unwind (ServerContext): registered USDT probes
1072025-04-30T06:42:51.751ZINFOtest_snapshot_unwind (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:42:51.886ZDEBGtest_snapshot_unwind (clickhouse-client): registered USDT probes id = 119e91ec-eb5b-4e69-903c-2f6306b28804
1092025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /bgtasks versions = all
1102025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /demo-saga versions = all
1202025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:42:51.964ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /metrics/producers versions = all
1442025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /mgs-updates versions = all
1452025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /sagas versions = all
1522025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /sleds/add versions = all
1572025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = GET path = /v1/ping versions = all
1612025-04-30T06:42:51.965ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:42:51.969ZDEBGtest_snapshot_unwind (dropshot_internal): registered endpoint local_addr = 127.0.0.1:53046 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:42:51.969ZINFOtest_snapshot_unwind (dropshot_internal): listening local_addr = 127.0.0.1:53046
1642025-04-30T06:42:51.970ZDEBGtest_snapshot_unwind (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:53046
1652025-04-30T06:42:51.971ZINFOtest_snapshot_unwind (SEC): SEC running sec_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
1662025-04-30T06:42:51.972ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:42:51.972ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:42:51.977ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:42:51.978ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in users
1702025-04-30T06:42:51.981ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:42:51.982ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:42:51.982ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:42:51.983ZINFOtest_snapshot_unwind (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:42:51.998ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: ad af 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:50183 req_id = 5a3cb083-dc54-49fa-ba6a-bd7ff4625c12
1752025-04-30T06:42:51.999ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44463,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50183 req_id = 5a3cb083-dc54-49fa-ba6a-bd7ff4625c12
1762025-04-30T06:42:52.001ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44463, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50183 req_id = 5a3cb083-dc54-49fa-ba6a-bd7ff4625c12
1772025-04-30T06:42:52.005ZINFOtest_snapshot_unwind (DataLoader): created 5 built-in users
1782025-04-30T06:42:52.005ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:42:52.005ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:42:52.008ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:42:52.009ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in roles
1822025-04-30T06:42:52.009ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:42:52.009ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:42:52.009ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:42:52.038ZINFOtest_snapshot_unwind (DataLoader): created 10 built-in roles
1862025-04-30T06:42:52.039ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:42:52.039ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:42:52.041ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:42:52.041ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:42:52.041ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:42:52.041ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:42:52.042ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:42:52.489ZINFOtest_snapshot_unwind (DataLoader): created 4 built-in role assignments
1942025-04-30T06:42:52.489ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:42:52.489ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:42:52.491ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:42:52.491ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in silos
1982025-04-30T06:42:52.491ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:42:52.491ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:42:52.492ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:42:52.607ZINFOtest_snapshot_unwind (DataLoader): created 2 built-in silos
2022025-04-30T06:42:52.607ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:42:52.607ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:42:52.608ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:42:52.621ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:42:52.621ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:42:52.623ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:42:52.623ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in projects
2092025-04-30T06:42:52.623ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:42:52.623ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:42:52.623ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:42:52.630ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:42:52.630ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:42:52.630ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:42:52.630ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:42:52.631ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:42:52.712ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:42:52.712ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:42:52.712ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:42:52.713ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:42:52.716ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:42:52.719ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:42:52.719ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:42:52.719ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:42:52.719ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:42:52.719ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:42:52.720ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:42:52.722ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:42:52.722ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:42:52.722ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:42:52.722ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:42:52.725ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:42:52.728ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:42:52.728ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:42:52.728ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:42:52.728ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:42:52.758ZINFOtest_snapshot_unwind (DataLoader): created built-in services project
2382025-04-30T06:42:52.758ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:42:52.758ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:42:52.760ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:42:52.760ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:42:52.760ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:42:52.760ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:42:52.760ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:42:52.763ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:42:52.763ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:42:52.764ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:42:52.768ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:42:52.768ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:42:52.768ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:42:52.768ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:42:52.768ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:42:52.770ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:42:52.770ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:42:52.770ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:42:52.771ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:42:52.773ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:42:52.773ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:42:52.773ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:42:52.773ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:42:52.775ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:42:52.780ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:42:52.780ZTRACtest_snapshot_unwind (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:42:52.780ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:42:52.780ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:42:52.780ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:42:52.780ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:42:52.781ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:42:52.783ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:42:52.783ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:42:52.784ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:42:52.784ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:42:52.788ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:42:52.788ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:42:52.788ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:42:52.788ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:42:52.804ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:42:52.809ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:42:52.809ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:42:52.809ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:42:52.810ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:42:52.823ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:42:52.823ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:42:52.823ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:42:52.823ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:42:52.823ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:42:52.826ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:42:52.826ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:42:52.826ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:42:52.826ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:42:52.828ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:42:52.828ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:42:52.828ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:42:52.829ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:42:52.831ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:42:52.838ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:42:52.839ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:42:52.839ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:42:52.839ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:42:52.849ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:42:52.849ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:42:52.850ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:42:52.859ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:42:52.860ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:42:52.860ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:42:52.864ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:42:52.864ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:42:52.864ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:42:52.864ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:42:52.865ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:42:52.867ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:42:52.867ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:42:52.867ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:42:52.867ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:42:52.869ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:42:52.869ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:42:52.869ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:42:52.870ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:42:52.872ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:42:52.878ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:42:52.878ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:42:52.878ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:42:52.879ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:42:52.951ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:42:52.951ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:42:52.951ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:42:52.951ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:42:52.952ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:42:52.954ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:42:52.954ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:42:52.954ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:42:52.955ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:42:52.957ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:42:52.957ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:42:52.957ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:42:52.958ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:42:52.959ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:42:52.969ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:42:52.969ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:42:52.969ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:42:52.969ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:42:53.072ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:42:53.072ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:42:53.072ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:42:53.072ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:42:53.073ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:42:53.076ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:42:53.076ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:42:53.076ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:42:53.076ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:42:53.079ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:42:53.079ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:42:53.079ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:42:53.079ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:42:53.081ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:42:53.098ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:42:53.098ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:42:53.098ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:42:53.099ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:42:53.110ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:42:53.111ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:42:53.111ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:42:53.112ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:42:53.124ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:42:53.124ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:42:53.125ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:42:53.136ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:42:53.136ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:42:53.137ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:42:53.145ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:42:53.145ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:42:53.145ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:42:53.145ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:42:53.146ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:42:53.150ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:42:53.150ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:42:53.150ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:42:53.150ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:42:53.187ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:42:53.191ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:42:53.194ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:42:53.197ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:42:53.203ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:42:53.213ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:42:53.213ZTRACtest_snapshot_unwind (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:42:53.213ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:42:53.213ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:42:53.214ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:42:53.214ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:42:53.214ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:42:53.221ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:42:53.221ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:42:53.221ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:42:53.222ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:42:53.237ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:42:53.238ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:42:53.239ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:42:53.239ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:42:53.268ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:42:53.286ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:42:53.287ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:42:53.287ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:42:53.287ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:42:53.318ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:42:53.318ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:42:53.319ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:42:53.319ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:42:53.320ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:42:53.326ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:42:53.326ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:42:53.326ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:42:53.327ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:42:53.333ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:42:53.333ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:42:53.333ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:42:53.334ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:42:53.342ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:42:53.354ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:42:53.354ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:42:53.355ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:42:53.355ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:42:53.530ZINFOtest_snapshot_unwind (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:42:53.530ZDEBGtest_snapshot_unwind (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:42:53.530ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:42:53.530ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:42:53.531ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:42:53.536ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:42:53.536ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:42:53.536ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:42:53.538ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:42:53.538ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:42:53.539ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:42:53.541ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:42:53.541ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:42:53.541ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:42:53.541ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:42:53.541ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:42:53.543ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:42:53.543ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:42:53.543ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:42:53.543ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:42:53.545ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:42:53.545ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:42:53.545ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:42:53.545ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:42:53.547ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:42:53.552ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:42:53.553ZTRACtest_snapshot_unwind (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:42:53.553ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:42:53.553ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:42:53.553ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:42:53.559ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:42:53.559ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:42:53.559ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:42:53.559ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:42:53.559ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:42:53.562ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:42:53.562ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:42:53.562ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:42:53.563ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:42:53.565ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:42:53.565ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:42:53.565ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:42:53.566ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:42:53.568ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:42:53.574ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:42:53.580ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:42:53.580ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:42:53.580ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:42:53.580ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:42:53.581ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:42:53.583ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:42:53.583ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:42:53.583ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:42:53.583ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:42:53.586ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:42:53.586ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:42:53.586ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:42:53.586ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:42:53.588ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:42:53.595ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:42:53.595ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:42:53.595ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:42:53.596ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:42:53.641ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:42:53.641ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:42:53.641ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:42:53.644ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:42:53.644ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:42:53.644ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:42:53.644ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:42:53.644ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:42:53.647ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:42:53.647ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:42:53.647ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:42:53.648ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:42:53.651ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:42:53.651ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:42:53.651ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:42:53.651ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:42:53.656ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:42:53.663ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:42:53.668ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:42:53.669ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:42:53.669ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:42:53.669ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:42:53.669ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:42:53.671ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:42:53.671ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:42:53.671ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:42:53.672ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:42:53.675ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:42:53.675ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:42:53.675ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:42:53.675ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:42:53.678ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:42:53.684ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:42:53.684ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:42:53.684ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:42:53.684ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:42:53.693ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:42:53.693ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:42:53.694ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:42:53.696ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:42:53.696ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:42:53.697ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:42:53.697ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:42:53.697ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:42:53.700ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:42:53.700ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:42:53.700ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:42:53.702ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:42:53.704ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:42:53.704ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:42:53.704ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:42:53.705ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:42:53.708ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:42:53.716ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:42:53.721ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:42:53.721ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:42:53.721ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:42:53.721ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:42:53.721ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:42:53.723ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:42:53.723ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:42:53.723ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:42:53.723ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:42:53.785ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:42:53.785ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:42:53.785ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:42:53.786ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:42:53.789ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:42:53.797ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:42:53.797ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:42:53.797ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:42:53.797ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:42:53.804ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:42:53.804ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:42:53.804ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:42:53.808ZINFOtest_snapshot_unwind (DataLoader): created built-in services vpc subnets
5612025-04-30T06:42:53.808ZINFOtest_snapshot_unwind (DataLoader): created built-in services vpc
5622025-04-30T06:42:53.808ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:42:53.808ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:42:53.810ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:42:53.810ZDEBGtest_snapshot_unwind (DataLoader): attempting to create silo users
5662025-04-30T06:42:53.812ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:42:53.812ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:42:53.812ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:42:53.830ZINFOtest_snapshot_unwind (DataLoader): created 2 silo users
5702025-04-30T06:42:53.830ZDEBGtest_snapshot_unwind (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:42:53.830ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:42:53.830ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:42:53.831ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:42:53.842ZINFOtest_snapshot_unwind (DataLoader): created 2 silo user role assignments
5752025-04-30T06:42:53.842ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:42:53.842ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:42:53.843ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:42:53.855ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:42:53.855ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:42:53.856ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:42:53.869ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:42:53.869ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:42:53.869ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:42:53.869ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:42:53.870ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:42:53.880ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:42:53.883ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:42:53.886ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:42:53.886ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:42:53.887ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:42:53.932ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:42:53.932ZTRACtest_snapshot_unwind (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:42:53.932ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:42:53.932ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:42:53.933ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:42:53.936ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:42:53.939ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:42:53.939ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:42:53.939ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:42:53.940ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:42:54.022ZTRACtest_snapshot_unwind (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:42:54.022ZDEBGtest_snapshot_unwind (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:42:54.024ZDEBGtest_snapshot_unwind (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:42:54.034ZINFOtest_snapshot_unwind (nexus): populate complete
6052025-04-30T06:42:54.034ZINFOtest_snapshot_unwind (nexus): activating background tasks
6062025-04-30T06:42:54.039ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_sled1
6072025-04-30T06:42:54.039ZINFOtest_snapshot_unwind (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:42:54.092ZINFOtest_snapshot_unwind (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:42:54.093ZINFOtest_snapshot_unwind (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:42:54.093ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:42:54.094ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:42:54.094ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:42:54.094ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:42:54.095ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:42:54.095ZWARNtest_snapshot_unwind (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 108.927\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:42:54.095ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:42:54.095ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:42:54.096ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:42:54.096ZWARNtest_snapshot_unwind (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 48.106\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:42:54.096ZINFOtest_snapshot_unwind (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:32:54.096547552 UTC
6322025-04-30T06:42:54.096ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:42:54.096ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:42:54.097ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:42:54.097ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:42:54.097ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:42:54.097ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:42:54.097ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:42:54.097ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:42:54.098ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:42:54.098ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:42:54.098ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:42:54.099ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:42:54.099ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:42:54.099ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:42:54.099ZWARNtest_snapshot_unwind (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:42:54.099ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 57.575\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:42:54.100ZWARNtest_snapshot_unwind (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 31.312\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:42:54.100ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:42:54.100ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 19.909\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:42:54.101ZWARNtest_snapshot_unwind (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 16.202\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:42:54.101ZINFOtest_snapshot_unwind (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 79.879\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:42:54.101ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:42:54.101ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:42:54.101ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:42:54.101ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:42:54.102ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:42:54.102ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:42:54.102ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:42:54.102ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:42:54.102ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:42:54.102ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:42:54.103ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:42:54.103ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:42:54.103ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:42:54.103ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:42:54.104ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:42:54.104ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:42:54.104ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:42:54.105ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:42:54.106ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:42:54.106ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:42:54.106ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:42:54.106ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:42:54.106ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:42:54.107ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:42:54.107ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:42:54.107ZINFOtest_snapshot_unwind (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:42:54.107ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = instance_updater elapsed = 337.432\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:42:54.107ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:42:54.107ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:42:54.107ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:42:54.108ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:42:54.108ZINFOtest_snapshot_unwind (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:42:54.108ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:42:54.108ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:42:54.108ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:42:54.108ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:42:54.109ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:42:54.109ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:42:54.109ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:42:54.109ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:42:54.109ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:42:54.110ZTRACtest_snapshot_unwind (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:42:54.110ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:42:54.110ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:42:54.111ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:42:54.111ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:42:54.111ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:42:54.111ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:42:54.111ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:42:54.111ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:42:54.111ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:42:54.112ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:42:54.112ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:42:54.112ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:42:54.112ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:42:54.112ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:42:54.112ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:42:54.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:42:54.113ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:42:54.113ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:42:54.113ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:42:54.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:42:54.113ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:42:54.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:42:54.114ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:42:54.114ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:42:54.114ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:42:54.114ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:42:54.114ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:42:54.115ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:42:54.115ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:42:54.115ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:42:54.116ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:42:54.116ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
7792025-04-30T06:42:54.116ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7802025-04-30T06:42:54.116ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
7812025-04-30T06:42:54.116ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7822025-04-30T06:42:54.117ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7832025-04-30T06:42:54.117ZINFOtest_snapshot_unwind (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpR2eM3T/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:42:54.117ZDEBGtest_snapshot_unwind (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpR2eM3T/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:42:54.117ZINFOtest_snapshot_unwind (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpWd0xB7/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7862025-04-30T06:42:54.117ZDEBGtest_snapshot_unwind (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpWd0xB7/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7872025-04-30T06:42:54.173ZDEBGtest_snapshot_unwind (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:63004 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:42:54.173ZINFOtest_snapshot_unwind (dropshot (Repo Depot)): listening local_addr = [::1]:63004 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:42:54.174ZDEBGtest_snapshot_unwind (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:63004 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:42:54.181ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:42:54.181ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:42:54.182ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:42:54.183ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:45714 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:42:54.184ZINFOtest_snapshot_unwind (dropshot): listening local_addr = [::1]:45714 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8602025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:45714 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8612025-04-30T06:42:54.184ZDEBGtest_snapshot_unwind (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8622025-04-30T06:42:54.185ZDEBGtest_snapshot_unwind (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:53046/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8632025-04-30T06:42:54.188ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 05 01 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:58006 req_id = c3ac82ad-3c68-4c63-bc00-3588ca0db2c5
8642025-04-30T06:42:54.188ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1281,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58006 req_id = c3ac82ad-3c68-4c63-bc00-3588ca0db2c5
8652025-04-30T06:42:54.188ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1281, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58006 req_id = c3ac82ad-3c68-4c63-bc00-3588ca0db2c5
8662025-04-30T06:42:54.188ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: e2 a5 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57366 req_id = 589b8f01-8713-4508-b0bd-780fcc07fc91
8672025-04-30T06:42:54.188ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58021,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57366 req_id = 589b8f01-8713-4508-b0bd-780fcc07fc91
8682025-04-30T06:42:54.189ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58021, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57366 req_id = 589b8f01-8713-4508-b0bd-780fcc07fc91
8692025-04-30T06:42:54.189ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8702025-04-30T06:42:54.189ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8712025-04-30T06:42:54.189ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8722025-04-30T06:42:54.189ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8732025-04-30T06:42:54.193ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-04-30T06:42:54.194ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:61497
8752025-04-30T06:42:54.194ZWARNtest_snapshot_unwind (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:42:54.194ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 99.69212ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:42:54.195ZWARNtest_snapshot_unwind (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:42:54.195ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 98.925743ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:42:54.198ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:61497 req_id = 6c172ccb-ebd3-4283-911b-affac60099a8 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:42:54.200ZINFOtest_snapshot_unwind (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:42:54.213ZINFOtest_snapshot_unwind (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:32:54.096547552 UTC pruned = {}
8822025-04-30T06:42:54.214ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 117.54923ms iteration = 1 status = {"expiration":"2025-04-30T06:32:54.096547552Z","pruned":[]}
8832025-04-30T06:42:54.232ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:42:54.233ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: de 4f 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .O..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64779 req_id = 6c3c1341-fc02-4071-ab25-5bce2e797e2f
8852025-04-30T06:42:54.233ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64779 req_id = 6c3c1341-fc02-4071-ab25-5bce2e797e2f
8862025-04-30T06:42:54.234ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56911, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64779 req_id = 6c3c1341-fc02-4071-ab25-5bce2e797e2f
8872025-04-30T06:42:54.236ZERROtest_snapshot_unwind (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8882025-04-30T06:42:54.236ZERROtest_snapshot_unwind (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8892025-04-30T06:42:54.236ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 138.187801ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8902025-04-30T06:42:54.306ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8912025-04-30T06:42:54.310ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
8922025-04-30T06:42:54.310ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
8932025-04-30T06:42:54.310ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
8942025-04-30T06:42:54.310ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
8952025-04-30T06:42:54.311ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8962025-04-30T06:42:54.312ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 213.353092ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8972025-04-30T06:42:54.314ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 63 bf 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 c............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53208 req_id = d06e3fb3-257a-479a-bfe8-ea28bc13fc63
8982025-04-30T06:42:54.314ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25535,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53208 req_id = d06e3fb3-257a-479a-bfe8-ea28bc13fc63
8992025-04-30T06:42:54.315ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25535, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53208 req_id = d06e3fb3-257a-479a-bfe8-ea28bc13fc63
9002025-04-30T06:42:54.315ZERROtest_snapshot_unwind (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9012025-04-30T06:42:54.315ZERROtest_snapshot_unwind (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9022025-04-30T06:42:54.315ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 216.984668ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9032025-04-30T06:42:54.317ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:42:54.321ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9052025-04-30T06:42:54.321ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9062025-04-30T06:42:54.321ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9072025-04-30T06:42:54.322ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9082025-04-30T06:42:54.333ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:42:54.365ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9102025-04-30T06:42:54.365ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9112025-04-30T06:42:54.365ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9122025-04-30T06:42:54.366ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9132025-04-30T06:42:54.388ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:42:54.412ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9152025-04-30T06:42:54.412ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9162025-04-30T06:42:54.412ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9172025-04-30T06:42:54.412ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9182025-04-30T06:42:54.412ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:42:54.422ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9202025-04-30T06:42:54.422ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9212025-04-30T06:42:54.422ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9222025-04-30T06:42:54.429ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9232025-04-30T06:42:54.439ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:42:54.462ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9252025-04-30T06:42:54.463ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9262025-04-30T06:42:54.463ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9272025-04-30T06:42:54.464ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9282025-04-30T06:42:54.521ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9292025-04-30T06:42:54.524ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9302025-04-30T06:42:54.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9312025-04-30T06:42:54.525ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9322025-04-30T06:42:54.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9332025-04-30T06:42:54.525ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9342025-04-30T06:42:54.525ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9352025-04-30T06:42:54.526ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9362025-04-30T06:42:54.526ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9372025-04-30T06:42:54.526ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9382025-04-30T06:42:54.607ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 497.059158ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9392025-04-30T06:42:54.690ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9402025-04-30T06:42:54.693ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9412025-04-30T06:42:54.693ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9422025-04-30T06:42:54.693ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9432025-04-30T06:42:54.694ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9442025-04-30T06:42:54.708ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9452025-04-30T06:42:54.708ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9462025-04-30T06:42:54.708ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9472025-04-30T06:42:54.722ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9482025-04-30T06:42:54.722ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9492025-04-30T06:42:54.723ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9502025-04-30T06:42:54.730ZDEBGtest_snapshot_unwind (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9512025-04-30T06:42:54.730ZINFOtest_snapshot_unwind (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9522025-04-30T06:42:54.730ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 621.477283ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9532025-04-30T06:42:54.742ZINFOtest_snapshot_unwind (SagaRecoverer): listed in-progress sagas count = 0
9542025-04-30T06:42:54.742ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 632.920783ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9552025-04-30T06:42:54.746ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9562025-04-30T06:42:54.746ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9572025-04-30T06:42:54.746ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9582025-04-30T06:42:54.750ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9592025-04-30T06:42:54.750ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9602025-04-30T06:42:54.751ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9612025-04-30T06:42:54.806ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 694.587971ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9622025-04-30T06:42:54.807ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9632025-04-30T06:42:54.812ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9642025-04-30T06:42:54.820ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9652025-04-30T06:42:54.825ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9662025-04-30T06:42:54.851ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 735.566829ms iteration = 1 status = {"by_rx":{},"error":null}
9672025-04-30T06:42:54.910ZWARNtest_snapshot_unwind (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9682025-04-30T06:42:54.910ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 811.122128ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9692025-04-30T06:42:54.910ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 797.614425ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9702025-04-30T06:42:54.919ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9712025-04-30T06:42:54.919ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9722025-04-30T06:42:54.919ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9732025-04-30T06:42:54.919ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9742025-04-30T06:42:54.920ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9752025-04-30T06:42:54.924ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9762025-04-30T06:42:54.924ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9772025-04-30T06:42:54.925ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b8 de 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56373 req_id = 4311b7e6-c531-46bb-8458-a5440377f95f
9782025-04-30T06:42:54.926ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47326,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56373 req_id = 4311b7e6-c531-46bb-8458-a5440377f95f
9792025-04-30T06:42:54.926ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47326, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56373 req_id = 4311b7e6-c531-46bb-8458-a5440377f95f
9802025-04-30T06:42:54.927ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9812025-04-30T06:42:54.927ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 827.274129ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9822025-04-30T06:42:54.931ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 816.707797ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9832025-04-30T06:42:54.937ZTRACtest_snapshot_unwind (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9842025-04-30T06:42:54.937ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9852025-04-30T06:42:54.937ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9862025-04-30T06:42:54.942ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9872025-04-30T06:42:55.081ZWARNtest_snapshot_unwind (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
9882025-04-30T06:42:55.081ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 987.687793ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
9892025-04-30T06:42:55.083ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9902025-04-30T06:42:55.088ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9912025-04-30T06:42:55.088ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9922025-04-30T06:42:55.088ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9932025-04-30T06:42:55.088ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9942025-04-30T06:42:55.088ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9952025-04-30T06:42:55.089ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9962025-04-30T06:42:55.097ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 995.19436ms iteration = 1 status = {}
9972025-04-30T06:42:55.098ZWARNtest_snapshot_unwind (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
9982025-04-30T06:42:55.098ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 1.002744854s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
9992025-04-30T06:42:55.101ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
10002025-04-30T06:42:55.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 999.441702ms iteration = 1 status = {"error":"inventory collection is None"}
10012025-04-30T06:42:55.103ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10022025-04-30T06:42:55.113ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10032025-04-30T06:42:55.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10042025-04-30T06:42:55.113ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10052025-04-30T06:42:55.116ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10062025-04-30T06:42:55.138ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
10072025-04-30T06:42:55.138ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10082025-04-30T06:42:55.150ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10092025-04-30T06:42:55.160ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10102025-04-30T06:42:55.160ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10112025-04-30T06:42:55.160ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10122025-04-30T06:42:55.192ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.08822107s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10132025-04-30T06:42:55.193ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10142025-04-30T06:42:55.193ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10152025-04-30T06:42:55.193ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10162025-04-30T06:42:55.193ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10172025-04-30T06:42:55.194ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10182025-04-30T06:42:55.211ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10192025-04-30T06:42:55.212ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10202025-04-30T06:42:55.212ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10212025-04-30T06:42:55.212ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10222025-04-30T06:42:55.212ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10232025-04-30T06:42:55.213ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10242025-04-30T06:42:55.213ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 1014938 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:61497 req_id = 6c172ccb-ebd3-4283-911b-affac60099a8 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10252025-04-30T06:42:55.214ZDEBGtest_snapshot_unwind (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:53046/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "6c172ccb-ebd3-4283-911b-affac60099a8", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10262025-04-30T06:42:55.214ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_crucible_pantry
10272025-04-30T06:42:55.215ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10282025-04-30T06:42:55.216ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10292025-04-30T06:42:55.216ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10302025-04-30T06:42:55.216ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10312025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10322025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10332025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10342025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10352025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10362025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10372025-04-30T06:42:55.217ZDEBGtest_snapshot_unwind (dropshot): registered endpoint kind = pantry local_addr = [::1]:61186 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10382025-04-30T06:42:55.217ZINFOtest_snapshot_unwind (dropshot): listening kind = pantry local_addr = [::1]:61186 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10392025-04-30T06:42:55.219ZDEBGtest_snapshot_unwind (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:61186 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10402025-04-30T06:42:55.219ZINFOtest_snapshot_unwind (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:61186 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10412025-04-30T06:42:55.219ZDEBGtest_snapshot_unwind (init_with_steps): Running step populate_internal_dns
10422025-04-30T06:42:55.219ZDEBGtest_snapshot_unwind: Populating Internal DNS
10432025-04-30T06:42:55.298ZINFOtest_snapshot_unwind: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:42:55.298220357Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "cef6f1f7-0bb4-4520-9f28-f372c582a644.host": [ Aaaa( ::1, ), ], "6d731a80-8693-4598-80e5-20fb1ab79f0f.host": [ Aaaa( ::1, ), ], "f7a56cc4-6855-4ee1-bf23-43c5f437915c.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 63004, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 37592, target: "6d731a80-8693-4598-80e5-20fb1ab79f0f.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 53046, target: "f7a56cc4-6855-4ee1-bf23-43c5f437915c.host.control-plane.oxide.internal", }, ), ], "92aa522e-103a-4436-b9f4-81bcad90f416.host": [ Aaaa( ::1, ), ], "6c4c48b5-9647-4b7a-ba06-9cebd7318846.host": [ Aaaa( ::1, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40326, target: "6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "872ffea5-8dc4-40a6-97ec-67f25ee7a7f9.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "872ffea5-8dc4-40a6-97ec-67f25ee7a7f9.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 64867, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 48618, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 42076, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "872ffea5-8dc4-40a6-97ec-67f25ee7a7f9.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 33617, target: "872ffea5-8dc4-40a6-97ec-67f25ee7a7f9.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 61186, target: "cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 54858, target: "92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal", }, ), ], "872ffea5-8dc4-40a6-97ec-67f25ee7a7f9.host": [ Aaaa( ::1, ), ], }, }, ], }
10442025-04-30T06:42:55.299ZDEBGtest_snapshot_unwind: client request body = Some(Body) method = PUT uri = http://[::1]:54858/config
10452025-04-30T06:42:55.304ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.200103829s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10462025-04-30T06:42:55.304ZINFOtest_snapshot_unwind (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10472025-04-30T06:42:55.304ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.19832273s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10482025-04-30T06:42:55.305ZINFOtest_snapshot_unwind (http): accepted connection kind = dns local_addr = [::1]:54858 remote_addr = [::1]:53624
10492025-04-30T06:42:55.311ZTRACtest_snapshot_unwind (http): incoming request kind = dns local_addr = [::1]:54858 method = PUT remote_addr = [::1]:53624 req_id = a6c41c31-b659-4d02-9383-46a6844b4861 uri = /config
10502025-04-30T06:42:55.312ZINFOtest_snapshot_unwind (store): attempting generation update new_generation = 1 req_id = a6c41c31-b659-4d02-9383-46a6844b4861
10512025-04-30T06:42:55.312ZINFOtest_snapshot_unwind (store): pruning trees for generations newer than 0
10522025-04-30T06:42:55.312ZDEBGtest_snapshot_unwind (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10532025-04-30T06:42:55.313ZDEBGtest_snapshot_unwind (store): updating current config
10542025-04-30T06:42:55.314ZDEBGtest_snapshot_unwind (store): flushing default tree
10552025-04-30T06:42:55.314ZINFOtest_snapshot_unwind (store): pruning trees for generations older than 1 keep = 3
10562025-04-30T06:42:55.314ZINFOtest_snapshot_unwind (store): updated generation new_generation = 1 req_id = a6c41c31-b659-4d02-9383-46a6844b4861
10572025-04-30T06:42:55.314ZINFOtest_snapshot_unwind (http): request completed kind = dns latency_us = 3770 local_addr = [::1]:54858 method = PUT remote_addr = [::1]:53624 req_id = a6c41c31-b659-4d02-9383-46a6844b4861 response_code = 204 uri = /config
10582025-04-30T06:42:55.315ZDEBGtest_snapshot_unwind: client response result = Ok(Response { url: "http://[::1]:54858/config", status: 204, headers: {"x-request-id": "a6c41c31-b659-4d02-9383-46a6844b4861", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
10592025-04-30T06:42:55.323ZDEBGtest_snapshot_unwind (init_with_steps): Running step configure_sled_agents
10602025-04-30T06:42:55.389ZDEBGtest_snapshot_unwind: client request body = Some(Body) method = PUT uri = http://[::1]:45714/omicron-config
10612025-04-30T06:42:55.392ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.280827155s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10622025-04-30T06:42:55.393ZTRACtest_snapshot_unwind (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10632025-04-30T06:42:55.394ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10642025-04-30T06:42:55.394ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10652025-04-30T06:42:55.395ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10662025-04-30T06:42:55.395ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10672025-04-30T06:42:55.395ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10682025-04-30T06:42:55.400ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10692025-04-30T06:42:55.403ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10702025-04-30T06:42:55.410ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10712025-04-30T06:42:55.410ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10722025-04-30T06:42:55.411ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10732025-04-30T06:42:55.411ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10742025-04-30T06:42:55.416ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:52585 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10752025-04-30T06:42:55.547ZINFOtest_snapshot_unwind (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
10762025-04-30T06:42:55.547ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:45714/vpc-routes
10772025-04-30T06:42:55.549ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10782025-04-30T06:42:55.558ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10792025-04-30T06:42:55.559ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10802025-04-30T06:42:55.559ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10812025-04-30T06:42:55.559ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10822025-04-30T06:42:55.564ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10832025-04-30T06:42:55.564ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10842025-04-30T06:42:55.565ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10852025-04-30T06:42:55.565ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10862025-04-30T06:42:55.565ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10872025-04-30T06:42:55.568ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:52585 req_id = e07dc99d-648e-4ee7-955b-5a16e669ac0d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10882025-04-30T06:42:55.568ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:39462 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10892025-04-30T06:42:55.579ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 10919 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:52585 req_id = e07dc99d-648e-4ee7-955b-5a16e669ac0d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10902025-04-30T06:42:55.607ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:39462 req_id = ce94d495-2c3b-448a-b89a-11b7f701f3aa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
10912025-04-30T06:42:55.611ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 4001 local_addr = [::1]:45714 method = GET remote_addr = [::1]:39462 req_id = ce94d495-2c3b-448a-b89a-11b7f701f3aa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
10922025-04-30T06:42:55.612ZDEBGtest_snapshot_unwind: client response result = Ok(Response { url: "http://[::1]:45714/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e07dc99d-648e-4ee7-955b-5a16e669ac0d", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
10932025-04-30T06:42:55.614ZDEBGtest_snapshot_unwind: client request body = Some(Body) method = PUT uri = http://[::1]:45714/network-bootstore-config
10942025-04-30T06:42:55.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45714/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce94d495-2c3b-448a-b89a-11b7f701f3aa", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
10952025-04-30T06:42:55.615ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10962025-04-30T06:42:55.615ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10972025-04-30T06:42:55.616ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10982025-04-30T06:42:55.617ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:52585 req_id = be343c8f-dc20-4744-9e27-2b84478878f2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10992025-04-30T06:42:55.621ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 4667 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:52585 req_id = be343c8f-dc20-4744-9e27-2b84478878f2 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
11002025-04-30T06:42:55.629ZTRACtest_snapshot_unwind (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11012025-04-30T06:42:55.630ZINFOtest_snapshot_unwind (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11022025-04-30T06:42:55.630ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 1.522989828s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11032025-04-30T06:42:55.630ZDEBGtest_snapshot_unwind: client response result = Ok(Response { url: "http://[::1]:45714/network-bootstore-config", status: 204, headers: {"x-request-id": "be343c8f-dc20-4744-9e27-2b84478878f2", "date": "Wed, 30 Apr 2025 06:42:55 GMT"} })
11042025-04-30T06:42:55.632ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_nexus_external
11052025-04-30T06:42:55.632ZDEBGtest_snapshot_unwind: Starting Nexus (external API)
11062025-04-30T06:42:58.376ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
11072025-04-30T06:42:58.376ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11082025-04-30T06:42:58.376ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11092025-04-30T06:42:58.376ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
11102025-04-30T06:42:58.376ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11112025-04-30T06:42:58.379ZTRACtest_snapshot_unwind (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
11122025-04-30T06:42:58.379ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 4.263942857s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
11132025-04-30T06:42:58.379ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11142025-04-30T06:42:58.379ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11152025-04-30T06:42:58.379ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11162025-04-30T06:42:58.379ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11172025-04-30T06:42:58.380ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11182025-04-30T06:42:58.383ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11192025-04-30T06:42:58.387ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11202025-04-30T06:42:58.387ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11212025-04-30T06:42:58.387ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11222025-04-30T06:42:58.387ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11232025-04-30T06:42:58.388ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.275923919s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11242025-04-30T06:42:58.388ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11252025-04-30T06:42:58.392ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11262025-04-30T06:42:58.392ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11272025-04-30T06:42:58.392ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11282025-04-30T06:42:58.392ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11292025-04-30T06:42:58.408ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11302025-04-30T06:42:58.408ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11312025-04-30T06:42:58.408ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11322025-04-30T06:42:58.408ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11332025-04-30T06:42:58.422ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11342025-04-30T06:42:58.427ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11352025-04-30T06:42:58.428ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11362025-04-30T06:42:58.428ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
11372025-04-30T06:42:58.429ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11382025-04-30T06:42:58.431ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11392025-04-30T06:42:58.441ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11402025-04-30T06:42:58.442ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11412025-04-30T06:42:58.442ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11422025-04-30T06:42:58.442ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11432025-04-30T06:42:58.442ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11442025-04-30T06:42:58.442ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11452025-04-30T06:42:58.459ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11462025-04-30T06:42:58.459ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11472025-04-30T06:42:58.465ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 31 f9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 1............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63851 req_id = 194cad35-7d9d-4d87-b5e2-caad2208990f
11482025-04-30T06:42:58.465ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12793,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63851 req_id = 194cad35-7d9d-4d87-b5e2-caad2208990f
11492025-04-30T06:42:58.465ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
11502025-04-30T06:42:58.465ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11512025-04-30T06:42:58.466ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63851 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 194cad35-7d9d-4d87-b5e2-caad2208990f
11522025-04-30T06:42:58.466ZWARNtest_snapshot_unwind (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11532025-04-30T06:42:58.467ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.369384603s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11542025-04-30T06:42:58.467ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11552025-04-30T06:42:58.467ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11562025-04-30T06:42:58.467ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11572025-04-30T06:42:58.468ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11582025-04-30T06:42:58.472ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4243, tv_nsec: 271383074 } })
11592025-04-30T06:42:58.472ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
11602025-04-30T06:42:58.473ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11612025-04-30T06:42:58.478ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11622025-04-30T06:42:58.478ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11632025-04-30T06:42:58.478ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11642025-04-30T06:42:58.479ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11652025-04-30T06:42:58.480ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11662025-04-30T06:42:58.480ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
11672025-04-30T06:42:58.481ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11682025-04-30T06:42:58.483ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11692025-04-30T06:42:58.493ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11702025-04-30T06:42:58.493ZTRACtest_snapshot_unwind (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11712025-04-30T06:42:58.493ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11722025-04-30T06:42:58.494ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11732025-04-30T06:42:58.494ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11742025-04-30T06:42:58.494ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11752025-04-30T06:42:58.494ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11762025-04-30T06:42:58.497ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5a 47 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ZG..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51771 req_id = 40c59e94-740e-44d5-a191-cd320e8da559
11772025-04-30T06:42:58.497ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23111,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51771 req_id = 40c59e94-740e-44d5-a191-cd320e8da559
11782025-04-30T06:42:58.498ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11792025-04-30T06:42:58.498ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51771 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 40c59e94-740e-44d5-a191-cd320e8da559
11802025-04-30T06:42:58.498ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11812025-04-30T06:42:58.498ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11822025-04-30T06:42:58.498ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11832025-04-30T06:42:58.498ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11842025-04-30T06:42:58.558ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
11852025-04-30T06:42:58.561ZINFOtest_snapshot_unwind (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11862025-04-30T06:42:58.561ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.460440947s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11872025-04-30T06:42:58.561ZINFOtest_snapshot_unwind (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
11882025-04-30T06:42:58.561ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:45714/eip-gateways
11892025-04-30T06:42:58.562ZINFOtest_snapshot_unwind (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
11902025-04-30T06:42:58.564ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11912025-04-30T06:42:58.564ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
11922025-04-30T06:42:58.564ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11932025-04-30T06:42:58.564ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:39462 req_id = 0a7e0444-20c7-4bcc-9d45-96e150fa7fb4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
11942025-04-30T06:42:58.566ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 2096 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:39462 req_id = 0a7e0444-20c7-4bcc-9d45-96e150fa7fb4 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
11952025-04-30T06:42:58.572ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 39 06 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 9............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52283 req_id = c2d1e721-63e0-4c10-9c92-f4febe0b4f05
11962025-04-30T06:42:58.572ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14598,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52283 req_id = c2d1e721-63e0-4c10-9c92-f4febe0b4f05
11972025-04-30T06:42:58.572ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
11982025-04-30T06:42:58.572ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14598, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52283 req_id = c2d1e721-63e0-4c10-9c92-f4febe0b4f05
11992025-04-30T06:42:58.572ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45714/eip-gateways", status: 204, headers: {"x-request-id": "0a7e0444-20c7-4bcc-9d45-96e150fa7fb4", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
12002025-04-30T06:42:58.574ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.465325459s iteration = 1 status = {}
12012025-04-30T06:42:58.578ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
12022025-04-30T06:42:58.578ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
12032025-04-30T06:42:58.578ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
12042025-04-30T06:42:58.652ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
12052025-04-30T06:42:58.652ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12062025-04-30T06:42:58.652ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12072025-04-30T06:42:58.652ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12082025-04-30T06:42:58.652ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12092025-04-30T06:42:58.653ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12102025-04-30T06:42:58.653ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12112025-04-30T06:42:58.653ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12122025-04-30T06:42:58.654ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12132025-04-30T06:42:58.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:45714/v2p
12142025-04-30T06:42:58.732ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12152025-04-30T06:42:58.736ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12162025-04-30T06:42:58.736ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12172025-04-30T06:42:58.736ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12182025-04-30T06:42:58.737ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12192025-04-30T06:42:58.737ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12202025-04-30T06:42:58.745ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12212025-04-30T06:42:58.745ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12222025-04-30T06:42:58.745ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12232025-04-30T06:42:58.746ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12242025-04-30T06:42:58.747ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12252025-04-30T06:42:58.747ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12262025-04-30T06:42:58.747ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12272025-04-30T06:42:58.747ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
12282025-04-30T06:42:58.748ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12292025-04-30T06:42:58.749ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:56034 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12302025-04-30T06:42:58.753ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:56034 req_id = 71e37d86-f519-425c-827b-929acd69465a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12312025-04-30T06:42:58.754ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1341 local_addr = [::1]:45714 method = GET remote_addr = [::1]:56034 req_id = 71e37d86-f519-425c-827b-929acd69465a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12322025-04-30T06:42:58.754ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
12332025-04-30T06:42:58.754ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12342025-04-30T06:42:58.754ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12352025-04-30T06:42:58.754ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12362025-04-30T06:42:58.755ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12372025-04-30T06:42:58.756ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12382025-04-30T06:42:58.756ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12392025-04-30T06:42:58.757ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12402025-04-30T06:42:58.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:45714/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "71e37d86-f519-425c-827b-929acd69465a", "content-length": "2", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
12412025-04-30T06:42:58.760ZINFOtest_snapshot_unwind (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12422025-04-30T06:42:58.761ZINFOtest_snapshot_unwind (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12432025-04-30T06:42:58.761ZINFOtest_snapshot_unwind (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12442025-04-30T06:42:58.762ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 4.659019622s iteration = 1 status = {}
12452025-04-30T06:42:58.763ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12462025-04-30T06:42:58.766ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12472025-04-30T06:42:58.766ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12482025-04-30T06:42:58.766ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12492025-04-30T06:42:58.767ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12502025-04-30T06:42:58.767ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12512025-04-30T06:42:58.767ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12522025-04-30T06:42:58.767ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12532025-04-30T06:42:58.768ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12542025-04-30T06:42:58.775ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12552025-04-30T06:42:58.775ZDEBGtest_snapshot_unwind (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12562025-04-30T06:42:58.775ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12572025-04-30T06:42:58.775ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
12582025-04-30T06:42:58.776ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12592025-04-30T06:42:58.790ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12602025-04-30T06:42:58.790ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12612025-04-30T06:42:58.792ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12622025-04-30T06:42:58.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:45714/artifacts-config
12632025-04-30T06:42:58.797ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:43987 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12642025-04-30T06:42:58.800ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac684150-8687-4483-946e-3c2f35d2ec8d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
12652025-04-30T06:42:58.801ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
12662025-04-30T06:42:58.802ZINFOtest_snapshot_unwind (InternalApi): Using static configuration for external switchports
12672025-04-30T06:42:58.802ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12682025-04-30T06:42:58.802ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
12692025-04-30T06:42:58.803ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12702025-04-30T06:42:58.803ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12712025-04-30T06:42:58.804ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12722025-04-30T06:42:58.804ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12732025-04-30T06:42:58.804ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12742025-04-30T06:42:58.804ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12752025-04-30T06:42:58.805ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:43987 req_id = 1583c761-b96a-431e-90fe-8f3414ae6078 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12762025-04-30T06:42:58.807ZINFOtest_snapshot_unwind (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpR2eM3T/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12772025-04-30T06:42:58.807ZDEBGtest_snapshot_unwind (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpR2eM3T/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12782025-04-30T06:42:58.807ZINFOtest_snapshot_unwind (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpWd0xB7/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12792025-04-30T06:42:58.807ZDEBGtest_snapshot_unwind (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpWd0xB7/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12802025-04-30T06:42:58.808ZINFOtest_snapshot_unwind (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpR2eM3T/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12812025-04-30T06:42:58.809ZINFOtest_snapshot_unwind (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpWd0xB7/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12822025-04-30T06:42:58.820ZINFOtest_snapshot_unwind (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12832025-04-30T06:42:58.821ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12842025-04-30T06:42:58.821ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12852025-04-30T06:42:58.821ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12862025-04-30T06:42:58.822ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12872025-04-30T06:42:58.822ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 16989 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:43987 req_id = 1583c761-b96a-431e-90fe-8f3414ae6078 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12882025-04-30T06:42:58.824ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12892025-04-30T06:42:58.824ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12902025-04-30T06:42:58.824ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12912025-04-30T06:42:58.825ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12922025-04-30T06:42:58.836ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:45714/artifacts-config", status: 204, headers: {"x-request-id": "1583c761-b96a-431e-90fe-8f3414ae6078", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
12932025-04-30T06:42:58.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:45714/artifacts
12942025-04-30T06:42:58.837ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12952025-04-30T06:42:58.844ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12962025-04-30T06:42:58.844ZTRACtest_snapshot_unwind (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
12972025-04-30T06:42:58.844ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12982025-04-30T06:42:58.844ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12992025-04-30T06:42:58.845ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13002025-04-30T06:42:58.847ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:43987 req_id = dbb2e124-0d7b-402e-9929-273c632c76e5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
13012025-04-30T06:42:58.848ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1361 local_addr = [::1]:45714 method = GET remote_addr = [::1]:43987 req_id = dbb2e124-0d7b-402e-9929-273c632c76e5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
13022025-04-30T06:42:58.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:45714/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbb2e124-0d7b-402e-9929-273c632c76e5", "content-length": "26", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13032025-04-30T06:42:58.856ZINFOtest_snapshot_unwind (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:45714
13042025-04-30T06:42:58.857ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13052025-04-30T06:42:58.857ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13062025-04-30T06:42:58.858ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13072025-04-30T06:42:58.870ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 4.756582059s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:58.837055885Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:42:58.856645466Z"}]}
13082025-04-30T06:42:58.875ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13092025-04-30T06:42:58.875ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13102025-04-30T06:42:58.876ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13112025-04-30T06:42:58.884ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc5d12f3-8f32-4da3-8378-44203bb29616", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13122025-04-30T06:42:58.887ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
13132025-04-30T06:42:58.888ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13142025-04-30T06:42:58.888ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13152025-04-30T06:42:58.890ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13162025-04-30T06:42:58.909ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13172025-04-30T06:42:58.909ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13182025-04-30T06:42:58.909ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13192025-04-30T06:42:58.909ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13202025-04-30T06:42:58.912ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13212025-04-30T06:42:58.915ZINFOtest_snapshot_unwind (InternalApi): Recording Rack Network Configuration
13222025-04-30T06:42:58.916ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13232025-04-30T06:42:58.916ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
13242025-04-30T06:42:58.916ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13252025-04-30T06:42:58.920ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:42:58.920ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:42:58.920ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:42:58.921ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:42:58.949ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3086752-7559-47d3-a6c7-b51d9823df75", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13302025-04-30T06:42:58.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
13312025-04-30T06:42:58.950ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13322025-04-30T06:42:58.950ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13332025-04-30T06:42:58.950ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13342025-04-30T06:42:58.951ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13352025-04-30T06:42:58.967ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13362025-04-30T06:42:58.980ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13372025-04-30T06:42:58.981ZTRACtest_snapshot_unwind (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13382025-04-30T06:42:58.981ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13392025-04-30T06:42:58.981ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13402025-04-30T06:42:58.981ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13412025-04-30T06:42:58.981ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13422025-04-30T06:42:58.984ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13432025-04-30T06:42:58.990ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13442025-04-30T06:42:58.990ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13452025-04-30T06:42:58.991ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13462025-04-30T06:42:58.991ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13472025-04-30T06:42:58.995ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13482025-04-30T06:42:58.995ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13492025-04-30T06:42:58.995ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13502025-04-30T06:42:58.996ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13512025-04-30T06:42:59.001ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13522025-04-30T06:42:59.010ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13532025-04-30T06:42:59.011ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13542025-04-30T06:42:59.011ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13552025-04-30T06:42:59.011ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13562025-04-30T06:42:59.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb16b029-fd4b-41ee-bdf1-e5ff1471303b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13572025-04-30T06:42:59.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
13582025-04-30T06:42:59.023ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13592025-04-30T06:42:59.023ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13602025-04-30T06:42:59.024ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13612025-04-30T06:42:59.028ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13622025-04-30T06:42:59.029ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13632025-04-30T06:42:59.030ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13642025-04-30T06:42:59.040ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13652025-04-30T06:42:59.040ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13662025-04-30T06:42:59.041ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13672025-04-30T06:42:59.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1162c71f-dc64-4732-9b14-b1d6b3eedf74", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13682025-04-30T06:42:59.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
13692025-04-30T06:42:59.048ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13702025-04-30T06:42:59.048ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13712025-04-30T06:42:59.048ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13722025-04-30T06:42:59.053ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13732025-04-30T06:42:59.053ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13742025-04-30T06:42:59.053ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13752025-04-30T06:42:59.053ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13762025-04-30T06:42:59.054ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13772025-04-30T06:42:59.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91dd36c7-23e8-4624-87cb-8ff305970929", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13782025-04-30T06:42:59.056ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
13792025-04-30T06:42:59.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6233d5cd-ac76-4902-a3f4-9737a9c3b851", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13802025-04-30T06:42:59.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
13812025-04-30T06:42:59.062ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f31f1c6-c328-4e6c-9fe2-4963d6d32301", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13822025-04-30T06:42:59.063ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
13832025-04-30T06:42:59.064ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4cf74e6-b97e-44a5-8949-9eb85c069f78", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13842025-04-30T06:42:59.064ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
13852025-04-30T06:42:59.066ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a077525d-e7fa-4090-a5f6-452d5d00e6f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13862025-04-30T06:42:59.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
13872025-04-30T06:42:59.068ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13882025-04-30T06:42:59.068ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13892025-04-30T06:42:59.068ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13902025-04-30T06:42:59.069ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13912025-04-30T06:42:59.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24ec92e3-0b1e-457b-93e1-110f90e7e71d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13922025-04-30T06:42:59.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
13932025-04-30T06:42:59.073ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13942025-04-30T06:42:59.073ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13952025-04-30T06:42:59.073ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13962025-04-30T06:42:59.073ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13972025-04-30T06:42:59.074ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bc79abf-398e-452a-9270-378b45a1eeb5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
13982025-04-30T06:42:59.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
13992025-04-30T06:42:59.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f0edf1f-4286-479d-858a-d7557480e818", "content-length": "734", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14002025-04-30T06:42:59.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
14012025-04-30T06:42:59.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f7fdd19-fdcf-407a-ae7f-f95087d27602", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14022025-04-30T06:42:59.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
14032025-04-30T06:42:59.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b28c5a66-d0f3-4554-9f34-2c6852004d82", "content-length": "111", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14042025-04-30T06:42:59.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
14052025-04-30T06:42:59.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "361dc0c2-7c84-498f-9180-f92bc1e5630d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14062025-04-30T06:42:59.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
14072025-04-30T06:42:59.091ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14082025-04-30T06:42:59.114ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14092025-04-30T06:42:59.114ZTRACtest_snapshot_unwind (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14102025-04-30T06:42:59.114ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14112025-04-30T06:42:59.114ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14122025-04-30T06:42:59.114ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14132025-04-30T06:42:59.115ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14142025-04-30T06:42:59.116ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14152025-04-30T06:42:59.132ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14162025-04-30T06:42:59.132ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14172025-04-30T06:42:59.134ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14182025-04-30T06:42:59.135ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14192025-04-30T06:42:59.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97613e93-c93e-42e6-86be-90347a9e32ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14202025-04-30T06:42:59.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
14212025-04-30T06:42:59.139ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14222025-04-30T06:42:59.139ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14232025-04-30T06:42:59.139ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14242025-04-30T06:42:59.139ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14252025-04-30T06:42:59.142ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14262025-04-30T06:42:59.174ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14272025-04-30T06:42:59.174ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14282025-04-30T06:42:59.174ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14292025-04-30T06:42:59.175ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14302025-04-30T06:42:59.176ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
14312025-04-30T06:42:59.176ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14322025-04-30T06:42:59.176ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14332025-04-30T06:42:59.176ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14342025-04-30T06:42:59.177ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14352025-04-30T06:42:59.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5fa066-51d0-4fbc-bfd8-6e64d59eed79", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14362025-04-30T06:42:59.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
14372025-04-30T06:42:59.182ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14382025-04-30T06:42:59.185ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
14392025-04-30T06:42:59.185ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14402025-04-30T06:42:59.186ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14412025-04-30T06:42:59.188ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14422025-04-30T06:42:59.193ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14432025-04-30T06:42:59.194ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14442025-04-30T06:42:59.194ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14452025-04-30T06:42:59.194ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14462025-04-30T06:42:59.194ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14472025-04-30T06:42:59.195ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14482025-04-30T06:42:59.196ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") }
14492025-04-30T06:42:59.196ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14502025-04-30T06:42:59.196ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14512025-04-30T06:42:59.196ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14522025-04-30T06:42:59.197ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14532025-04-30T06:42:59.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "374af784-c240-41ac-9584-862b214ef2e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14542025-04-30T06:42:59.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
14552025-04-30T06:42:59.199ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14562025-04-30T06:42:59.203ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14572025-04-30T06:42:59.206ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14582025-04-30T06:42:59.206ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14592025-04-30T06:42:59.207ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14602025-04-30T06:42:59.207ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14612025-04-30T06:42:59.211ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14622025-04-30T06:42:59.212ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14632025-04-30T06:42:59.212ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14642025-04-30T06:42:59.212ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14652025-04-30T06:42:59.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbffdbc6-21f9-4e42-b285-22bf7b9a546f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14662025-04-30T06:42:59.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
14672025-04-30T06:42:59.217ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") }
14682025-04-30T06:42:59.217ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14692025-04-30T06:42:59.217ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14702025-04-30T06:42:59.217ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14712025-04-30T06:42:59.217ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14722025-04-30T06:42:59.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f100665-ff2e-41c6-a63e-0e2f5aa00c8e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14732025-04-30T06:42:59.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
14742025-04-30T06:42:59.218ZTRACtest_snapshot_unwind (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14752025-04-30T06:42:59.218ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14762025-04-30T06:42:59.218ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14772025-04-30T06:42:59.218ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14782025-04-30T06:42:59.221ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14792025-04-30T06:42:59.223ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14802025-04-30T06:42:59.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ffd1c10-f577-494e-8123-82e20e15bdfd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14812025-04-30T06:42:59.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
14822025-04-30T06:42:59.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "961c30bb-18c7-48e7-bd3a-57d5e66c8d3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14832025-04-30T06:42:59.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
14842025-04-30T06:42:59.228ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bb1c601-510a-4724-a29e-2225df60a9d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14852025-04-30T06:42:59.228ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
14862025-04-30T06:42:59.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32fb5bcd-981b-4939-858c-01d0b8140971", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14872025-04-30T06:42:59.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
14882025-04-30T06:42:59.233ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") }
14892025-04-30T06:42:59.233ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14902025-04-30T06:42:59.233ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14912025-04-30T06:42:59.234ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
14922025-04-30T06:42:59.234ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14932025-04-30T06:42:59.236ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14942025-04-30T06:42:59.240ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: cb407844-c935-49e5-96ba-bda332ffaf2a, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14952025-04-30T06:42:59.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d065a7f8-5ee6-43a5-b6ab-427e943140ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14962025-04-30T06:42:59.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
14972025-04-30T06:42:59.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f261992-8a1d-4f32-af51-d18bece1cfa4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
14982025-04-30T06:42:59.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
14992025-04-30T06:42:59.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c11e70-7baf-4fc5-ad35-b6fbd6dd19aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15002025-04-30T06:42:59.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
15012025-04-30T06:42:59.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a400a175-44a5-45ff-8508-6f42e9cbbfdf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15022025-04-30T06:42:59.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
15032025-04-30T06:42:59.259ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "057ee024-068e-4c7e-bc90-3f8784ea8d72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15042025-04-30T06:42:59.259ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
15052025-04-30T06:42:59.261ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15062025-04-30T06:42:59.261ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15072025-04-30T06:42:59.261ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15082025-04-30T06:42:59.261ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
15092025-04-30T06:42:59.262ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15102025-04-30T06:42:59.269ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15112025-04-30T06:42:59.273ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15122025-04-30T06:42:59.277ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d746a35a-1a59-4139-9e3b-780baf26b5c4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15132025-04-30T06:42:59.277ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
15142025-04-30T06:42:59.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a59ccd44-d5e8-42c2-b479-bb5cb36e805e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15152025-04-30T06:42:59.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
15162025-04-30T06:42:59.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7ea077-7376-4462-894c-ee18170a3663", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15172025-04-30T06:42:59.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
15182025-04-30T06:42:59.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e12e6c-b4af-48dd-9f83-dd6df8b2155d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15192025-04-30T06:42:59.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
15202025-04-30T06:42:59.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a813c33-c49d-49e4-8572-91e2858923d6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15212025-04-30T06:42:59.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
15222025-04-30T06:42:59.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27565a1c-ad53-4ab9-9b0d-82ad147e6287", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15232025-04-30T06:42:59.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
15242025-04-30T06:42:59.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b78b39f4-e1bc-4f57-bddf-04d44810b048", "content-length": "109", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15252025-04-30T06:42:59.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
15262025-04-30T06:42:59.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da2013f4-1b01-4d13-89c4-2e517625f358", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15272025-04-30T06:42:59.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
15282025-04-30T06:42:59.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ccd039-2145-46c9-a844-49f7d5b2679a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15292025-04-30T06:42:59.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
15302025-04-30T06:42:59.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bf9aae4-d724-4458-83cc-2f2cdf0cb6ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15312025-04-30T06:42:59.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
15322025-04-30T06:42:59.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4c21f11-4641-4da3-ac5c-efda64274cb9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15332025-04-30T06:42:59.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
15342025-04-30T06:42:59.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f3e13b-6283-48f8-bdb8-c8003967add5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15352025-04-30T06:42:59.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
15362025-04-30T06:42:59.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db35906-e056-4295-9941-1982268731d3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15372025-04-30T06:42:59.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
15382025-04-30T06:42:59.341ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf4921bf-395e-4716-8a3c-80c5621dd23b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15392025-04-30T06:42:59.341ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
15402025-04-30T06:42:59.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7958648c-f413-4aa5-ae8f-12cc5edbe0c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:42:58 GMT"} })
15412025-04-30T06:42:59.342ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15422025-04-30T06:42:59.342ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15432025-04-30T06:42:59.342ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15442025-04-30T06:42:59.344ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15452025-04-30T06:42:59.345ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15462025-04-30T06:42:59.351ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15472025-04-30T06:42:59.355ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15482025-04-30T06:42:59.355ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15492025-04-30T06:42:59.356ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15502025-04-30T06:42:59.356ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15512025-04-30T06:42:59.356ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15522025-04-30T06:42:59.356ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15532025-04-30T06:42:59.369ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-04-30T06:42:59.374ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15552025-04-30T06:42:59.374ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15562025-04-30T06:42:59.374ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15572025-04-30T06:42:59.375ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15582025-04-30T06:42:59.516ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15592025-04-30T06:42:59.516ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15602025-04-30T06:42:59.516ZINFOtest_snapshot_unwind (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15612025-04-30T06:42:59.516ZDEBGtest_snapshot_unwind (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.408150046s
15622025-04-30T06:42:59.516ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.408201458s iteration = 1 status = {}
15632025-04-30T06:42:59.529ZINFOtest_snapshot_unwind (InternalApi): inserted blueprint blueprint_id = 856aa7e3-0939-484c-b06c-c418570c76fd
15642025-04-30T06:42:59.529ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15652025-04-30T06:42:59.529ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15662025-04-30T06:42:59.529ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15672025-04-30T06:42:59.529ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
15682025-04-30T06:42:59.530ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15692025-04-30T06:42:59.532ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15702025-04-30T06:42:59.535ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15712025-04-30T06:42:59.584ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
15722025-04-30T06:42:59.584ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
15732025-04-30T06:42:59.585ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:56102 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15742025-04-30T06:42:59.585ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:56102 req_id = da569bf3-de12-4885-884f-b503eff1a28c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15752025-04-30T06:42:59.586ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 553 local_addr = [::1]:45714 method = GET remote_addr = [::1]:56102 req_id = da569bf3-de12-4885-884f-b503eff1a28c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15762025-04-30T06:42:59.586ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "da569bf3-de12-4885-884f-b503eff1a28c", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
15772025-04-30T06:42:59.587ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15782025-04-30T06:42:59.587ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15792025-04-30T06:42:59.588ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
15802025-04-30T06:42:59.589ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15812025-04-30T06:42:59.589ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15822025-04-30T06:42:59.589ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15832025-04-30T06:42:59.589ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15842025-04-30T06:42:59.590ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15852025-04-30T06:42:59.592ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15862025-04-30T06:42:59.596ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15872025-04-30T06:42:59.597ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15882025-04-30T06:42:59.597ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15892025-04-30T06:42:59.597ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15902025-04-30T06:42:59.609ZINFOtest_snapshot_unwind (InternalApi): Inserted networking records for external_dns service
15912025-04-30T06:42:59.609ZINFOtest_snapshot_unwind (InternalApi): No networking records needed for cockroach_db service
15922025-04-30T06:42:59.609ZINFOtest_snapshot_unwind (InternalApi): No networking records needed for clickhouse service
15932025-04-30T06:42:59.609ZINFOtest_snapshot_unwind (InternalApi): No networking records needed for internal_dns service
15942025-04-30T06:42:59.609ZINFOtest_snapshot_unwind (InternalApi): No networking records needed for crucible_pantry service
15952025-04-30T06:42:59.636ZINFOtest_snapshot_unwind (InternalApi): Inserted networking records for nexus service
15962025-04-30T06:42:59.636ZINFOtest_snapshot_unwind (InternalApi): Inserted service networking records
15972025-04-30T06:42:59.636ZINFOtest_snapshot_unwind (InternalApi): Inserted physical disks
15982025-04-30T06:42:59.636ZINFOtest_snapshot_unwind (InternalApi): Inserted zpools
15992025-04-30T06:42:59.636ZINFOtest_snapshot_unwind (InternalApi): Inserted datasets
16002025-04-30T06:42:59.655ZINFOtest_snapshot_unwind (InternalApi): Populated DNS tables for internal DNS
16012025-04-30T06:42:59.659ZINFOtest_snapshot_unwind (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:42:59.660ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:42:59.660ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:42:59.660ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:42:59.660ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:42:59.660ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:42:59.663ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:42:59.666ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:42:59.666ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:42:59.666ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:42:59.666ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:42:59.666ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:42:59.666ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:42:59.668ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:42:59.671ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:42:59.723ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:42:59.723ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:42:59.723ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:42:59.723ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:42:59.724ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:42:59.730ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:42:59.737ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:42:59.737ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:42:59.737ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:42:59.738ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:42:59.738ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:42:59.739ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:42:59.750ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:42:59.756ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:42:59.759ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:42:59.759ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:42:59.759ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:42:59.759ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:42:59.759ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:42:59.774ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:42:59.783ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:42:59.807ZINFOtest_snapshot_unwind (InternalApi): Created recovery silo
16382025-04-30T06:42:59.809ZINFOtest_snapshot_unwind (InternalApi): Created recovery user
16392025-04-30T06:42:59.813ZINFOtest_snapshot_unwind (InternalApi): Created recovery user's password
16402025-04-30T06:42:59.813ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 9587148f-d9a3-4861-a834-bcd2b615e746, lookup_type: ById(9587148f-d9a3-4861-a834-bcd2b615e746) }
16412025-04-30T06:42:59.813ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 9587148f-d9a3-4861-a834-bcd2b615e746 resource_type = Silo
16422025-04-30T06:42:59.813ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:42:59.813ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:42:59.814ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:42:59.816ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:42:59.816ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:42:59.816ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:42:59.816ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:42:59.827ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:42:59.830ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 9587148f-d9a3-4861-a834-bcd2b615e746, lookup_type: ById(9587148f-d9a3-4861-a834-bcd2b615e746) } result = Ok(())
16512025-04-30T06:42:59.831ZDEBGtest_snapshot_unwind (InternalApi): Generated role assignment queries
16522025-04-30T06:42:59.838ZINFOtest_snapshot_unwind (InternalApi): Granted Silo privileges
16532025-04-30T06:42:59.838ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:42:59.838ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:42:59.838ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:42:59.838ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:42:59.839ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:42:59.844ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:42:59.847ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:42:59.867ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:42:59.867ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:42:59.869ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:42:59.874ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:42:59.874ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:42:59.875ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:42:59.879ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:42:59.879ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:42:59.880ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:42:59.888ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:42:59.888ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:42:59.888ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:42:59.888ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:42:59.889ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:42:59.892ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:42:59.892ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:42:59.892ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:42:59.893ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:42:59.896ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:42:59.896ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:42:59.896ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:42:59.897ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:42:59.903ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:42:59.912ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:42:59.913ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:42:59.913ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:42:59.913ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:42:59.913ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:42:59.913ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:42:59.914ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:42:59.919ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:42:59.919ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:42:59.919ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:42:59.919ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:42:59.922ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:42:59.923ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:42:59.923ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:42:59.923ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:42:59.926ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:42:59.934ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:42:59.934ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:42:59.934ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:42:59.934ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:42:59.943ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:42:59.943ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:42:59.944ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:42:59.948ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:42:59.948ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:42:59.949ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:42:59.978ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:42:59.978ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:42:59.979ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:42:59.985ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:42:59.985ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:42:59.985ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:42:59.985ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:42:59.987ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:42:59.991ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:42:59.991ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:42:59.991ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:42:59.992ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:42:59.996ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:42:59.996ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:42:59.996ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:42:59.997ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:43:00.002ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:43:00.021ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:43:00.021ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:43:00.021ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:43:00.021ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:43:00.022ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:43:00.026ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:43:00.026ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:43:00.028ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:43:00.030ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:43:00.030ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:43:00.031ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:43:00.033ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:43:00.033ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:43:00.044ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:43:00.054ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:43:00.056ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:43:00.056ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:43:00.056ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:43:00.059ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:43:00.068ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:43:00.068ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:43:00.068ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:43:00.070ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:43:00.071ZINFOtest_snapshot_unwind (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 7cce6308-ea2f-4bdc-bd6e-8cdb01380105
17502025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 7cce6308-ea2f-4bdc-bd6e-8cdb01380105 time_started = 2025-04-30 06:42:58.577925 UTC
17512025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 4.860345331s iteration = 2 status = {"collection_id":"7cce6308-ea2f-4bdc-bd6e-8cdb01380105","time_done":"2025-04-30 06:42:59.587900 UTC","time_started":"2025-04-30 06:42:58.577925 UTC"}
17522025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17532025-04-30T06:43:00.071ZWARNtest_snapshot_unwind (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17542025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 25.561\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17552025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17562025-04-30T06:43:00.071ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 9.71\u{b5}s iteration = 2 status = {"error":"task disabled"}
17572025-04-30T06:43:00.080ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17582025-04-30T06:43:00.080ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:43:00.080ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:43:00.080ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:43:00.088ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17622025-04-30T06:43:00.097ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17632025-04-30T06:43:00.097ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17642025-04-30T06:43:00.097ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17652025-04-30T06:43:00.097ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17662025-04-30T06:43:00.097ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:43:00.097ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:43:00.098ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:43:00.104ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:43:00.104ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:43:00.104ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:43:00.104ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:43:00.114ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:43:00.114ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:43:00.114ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:43:00.115ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:43:00.120ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:43:00.127ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17792025-04-30T06:43:00.127ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:43:00.127ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:43:00.128ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:43:00.134ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:43:00.134ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:43:00.135ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:43:00.138ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:43:00.138ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:43:00.139ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:43:00.141ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:43:00.141ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:43:00.141ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:43:00.145ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:43:00.145ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:43:00.145ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:43:00.149ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:43:00.149ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:43:00.149ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:43:00.149ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:43:00.150ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:43:00.153ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:43:00.153ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:43:00.153ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:43:00.154ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:43:00.158ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:43:00.158ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:43:00.160ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:43:00.162ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:43:00.168ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:43:00.181ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:43:00.181ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:43:00.181ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:43:00.181ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:43:00.181ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:43:00.181ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:43:00.181ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:43:00.197ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:43:00.197ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:43:00.197ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:43:00.197ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:43:00.209ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:43:00.209ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:43:00.209ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:43:00.211ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:43:00.233ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:43:00.243ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:43:00.243ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:43:00.243ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:43:00.243ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:43:00.257ZDEBGtest_snapshot_unwind (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 581ba028-07c9-4415-b60d-b5925e41fb0a, kind: Service { id: f7a56cc4-6855-4ee1-bf23-43c5f437915c }, name: Name("nexus-f7a56cc4-6855-4ee1-bf23-43c5f437915c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 03c0792c-5eb3-47a6-ba0a-3459a993795f, kind: Service { id: 6c4c48b5-9647-4b7a-ba06-9cebd7318846 }, name: Name("external-dns-6c4c48b5-9647-4b7a-ba06-9cebd7318846"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:43:00.258ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:43:00.258ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:43:00.258ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:43:00.258ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:43:00.258ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:43:00.287ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:43:00.293ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:43:00.293ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:43:00.293ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:43:00.293ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:43:00.302ZTRACtest_snapshot_unwind (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:59.847845Z, allowed_ips: None }
18402025-04-30T06:43:00.309ZDEBGtest_snapshot_unwind (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:43:00.310ZDEBGtest_snapshot_unwind (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 03c0792c-5eb3-47a6-ba0a-3459a993795f, kind: Service { id: 6c4c48b5-9647-4b7a-ba06-9cebd7318846 }, name: Name("external-dns-6c4c48b5-9647-4b7a-ba06-9cebd7318846"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 581ba028-07c9-4415-b60d-b5925e41fb0a, kind: Service { id: f7a56cc4-6855-4ee1-bf23-43c5f437915c }, name: Name("nexus-f7a56cc4-6855-4ee1-bf23-43c5f437915c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:43:00.310ZDEBGtest_snapshot_unwind (nexus): resolved 2 rules for sleds
18432025-04-30T06:43:00.423ZDEBGtest_snapshot_unwind (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:54.200591Z, time_modified: 2025-04-30T06:42:54.200591Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(45714), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(63004) }]
18442025-04-30T06:43:00.424ZDEBGtest_snapshot_unwind (nexus): sending firewall rules to sled agents
18452025-04-30T06:43:00.424ZTRACtest_snapshot_unwind (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:43:00.424ZDEBGtest_snapshot_unwind (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:43:00.425ZDEBGtest_snapshot_unwind (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:43:00.431ZTRACtest_snapshot_unwind (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:43:00.431ZTRACtest_snapshot_unwind (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:43:00.431ZTRACtest_snapshot_unwind (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:43:00.431ZDEBGtest_snapshot_unwind (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:43:00.432ZDEBGtest_snapshot_unwind (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:43:00.434ZDEBGtest_snapshot_unwind (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:43:00.438ZDEBGtest_snapshot_unwind (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:43:00.489ZDEBGtest_snapshot_unwind (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:45714/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:43:00.491ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:36092 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:43:00.494ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:36092 req_id = 720ae5c5-3b11-49a8-ae82-e6f8e04ccd58 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:43:00.497ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 3219 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:36092 req_id = 720ae5c5-3b11-49a8-ae82-e6f8e04ccd58 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:43:00.497ZDEBGtest_snapshot_unwind (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:45714/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "720ae5c5-3b11-49a8-ae82-e6f8e04ccd58", "date": "Wed, 30 Apr 2025 06:42:59 GMT"} })
18602025-04-30T06:43:00.507ZINFOtest_snapshot_unwind (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:43:00.507ZTRACtest_snapshot_unwind (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:43:00.507ZDEBGtest_snapshot_unwind (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:43:00.512ZDEBGtest_snapshot_unwind (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:43:00.512ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:43:00.512ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:43:00.512ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:43:00.512ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:43:00.513ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:43:00.513ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:43:00.513ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:43:00.513ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:43:00.513ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:43:00.513ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:43:00.514ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:43:00.514ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:43:00.514ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:43:00.514ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:43:00.515ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:43:00.515ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:43:00.515ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:43:00.515ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:43:00.515ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:43:00.516ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:43:00.516ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:43:00.516ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:43:00.516ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:43:00.516ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:43:00.516ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:43:00.517ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:43:00.518ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: ee ed 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:59551 req_id = 8629eb2a-7cd6-4e8a-b333-67627bf440ce
18912025-04-30T06:43:00.518ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61165,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59551 req_id = 8629eb2a-7cd6-4e8a-b333-67627bf440ce
18922025-04-30T06:43:00.518ZDEBGtest_snapshot_unwind (store): query key key = _nameservice._tcp
18932025-04-30T06:43:00.519ZDEBGtest_snapshot_unwind (store): query key key = 92aa522e-103a-4436-b9f4-81bcad90f416.host
18942025-04-30T06:43:00.519ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59551 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 54858, target: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal") })) }] req_id = 8629eb2a-7cd6-4e8a-b333-67627bf440ce
18952025-04-30T06:43:00.519ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: e3 03 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55118 req_id = 17d8fcc1-f863-4485-8cda-f5eef84c0b19
18962025-04-30T06:43:00.519ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58115,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55118 req_id = 17d8fcc1-f863-4485-8cda-f5eef84c0b19
18972025-04-30T06:43:00.519ZDEBGtest_snapshot_unwind (store): query key key = _external-dns._tcp
18982025-04-30T06:43:00.520ZDEBGtest_snapshot_unwind (store): query key key = 6c4c48b5-9647-4b7a-ba06-9cebd7318846.host
18992025-04-30T06:43:00.520ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55118 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40326, target: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal") })) }] req_id = 17d8fcc1-f863-4485-8cda-f5eef84c0b19
19002025-04-30T06:43:00.521ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 54858, target: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 320607640 } })
19012025-04-30T06:43:00.521ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:43:00.522ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40326, target: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 321323577 } })
19032025-04-30T06:43:00.522ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:43:00.522ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:43:00.522ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:43:00.523ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:43:00.523ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:43:00.523ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:43:00.523ZTRACtest_snapshot_unwind (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:43:00.524ZTRACtest_snapshot_unwind (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:43:00.524ZTRACtest_snapshot_unwind (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:43:00.524ZDEBGtest_snapshot_unwind (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:43:00.524ZDEBGtest_snapshot_unwind (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:43:00.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:43:00.525ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:43:00.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:43:00.525ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:43:00.526ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:43:00.526ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 2b 9d 01 00 00 01 00 00 00 00 00 01 24 39 32 61 +...........$92a\n0010: 61 35 32 32 65 2d 31 30 33 61 2d 34 34 33 36 2d a522e-103a-4436-\n0020: 62 39 66 34 2d 38 31 62 63 61 64 39 30 66 34 31 b9f4-81bcad90f41\n0030: 36 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 6.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:52176 req_id = b7958143-e8a7-4948-b72f-a27e8b0028c9
19202025-04-30T06:43:00.527ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11165,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 57,\n 50,\n 97,\n 97,\n 53,\n 50,\n 50,\n 101,\n 45,\n 49,\n 48,\n 51,\n 97,\n 45,\n 52,\n 52,\n 51,\n 54,\n 45,\n 98,\n 57,\n 102,\n 52,\n 45,\n 56,\n 49,\n 98,\n 99,\n 97,\n 100,\n 57,\n 48,\n 102,\n 52,\n 49,\n 54,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52176 req_id = b7958143-e8a7-4948-b72f-a27e8b0028c9
19212025-04-30T06:43:00.527ZDEBGtest_snapshot_unwind (store): query key key = 92aa522e-103a-4436-b9f4-81bcad90f416.host
19222025-04-30T06:43:00.527ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52176 query = LowerQuery { name: LowerName(Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal.")), original: Query { name: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("92aa522e-103a-4436-b9f4-81bcad90f416.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b7958143-e8a7-4948-b72f-a27e8b0028c9
19232025-04-30T06:43:00.527ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 71 ce 01 00 00 01 00 00 00 00 00 01 24 36 63 34 q...........$6c4\n0010: 63 34 38 62 35 2d 39 36 34 37 2d 34 62 37 61 2d c48b5-9647-4b7a-\n0020: 62 61 30 36 2d 39 63 65 62 64 37 33 31 38 38 34 ba06-9cebd731884\n0030: 36 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 6.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:49364 req_id = fb5b8d8f-60de-4a13-8002-94b52d86a537
19242025-04-30T06:43:00.528ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29134,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 54,\n 99,\n 52,\n 99,\n 52,\n 56,\n 98,\n 53,\n 45,\n 57,\n 54,\n 52,\n 55,\n 45,\n 52,\n 98,\n 55,\n 97,\n 45,\n 98,\n 97,\n 48,\n 54,\n 45,\n 57,\n 99,\n 101,\n 98,\n 100,\n 55,\n 51,\n 49,\n 56,\n 56,\n 52,\n 54,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49364 req_id = fb5b8d8f-60de-4a13-8002-94b52d86a537
19252025-04-30T06:43:00.528ZDEBGtest_snapshot_unwind (store): query key key = 6c4c48b5-9647-4b7a-ba06-9cebd7318846.host
19262025-04-30T06:43:00.528ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49364 query = LowerQuery { name: LowerName(Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal.")), original: Query { name: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("6c4c48b5-9647-4b7a-ba06-9cebd7318846.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fb5b8d8f-60de-4a13-8002-94b52d86a537
19272025-04-30T06:43:00.530ZINFOtest_snapshot_unwind (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:54858] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:43:00.530ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 17.075662ms iteration = 2 status = {"addresses":["[::1]:54858"]}
19292025-04-30T06:43:00.530ZINFOtest_snapshot_unwind (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:40326] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:43:00.530ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 16.21147ms iteration = 2 status = {"addresses":["[::1]:40326"]}
19312025-04-30T06:43:00.531ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:43:00.531ZWARNtest_snapshot_unwind (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:43:00.532ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 985.293\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:43:00.532ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:43:00.532ZWARNtest_snapshot_unwind (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:43:00.532ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 250.278\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:43:00.533ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:43:00.536ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:43:00.536ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:43:00.537ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:43:00.537ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:43:00.537ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:43:00.566ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:43:00.566ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:43:00.566ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:43:00.566ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:43:00.566ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:43:00.574ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:43:00.576ZDEBGtest_snapshot_unwind (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19502025-04-30T06:43:00.576ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19512025-04-30T06:43:00.576ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19522025-04-30T06:43:00.576ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19532025-04-30T06:43:00.576ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19542025-04-30T06:43:00.576ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19552025-04-30T06:43:00.576ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19562025-04-30T06:43:00.577ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19572025-04-30T06:43:00.579ZDEBGtest_snapshot_unwind (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-04-30T06:43:00.584ZDEBGtest_snapshot_unwind (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19592025-04-30T06:43:00.584ZINFOtest_snapshot_unwind (nexus): Rack initialized
19602025-04-30T06:43:00.584ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19612025-04-30T06:43:00.584ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
19622025-04-30T06:43:00.584ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19632025-04-30T06:43:00.585ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19642025-04-30T06:43:00.592ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19652025-04-30T06:43:00.593ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19662025-04-30T06:43:00.593ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19672025-04-30T06:43:00.593ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19682025-04-30T06:43:00.593ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19692025-04-30T06:43:00.597ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19702025-04-30T06:43:00.606ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
19712025-04-30T06:43:00.606ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19722025-04-30T06:43:00.618ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-04-30T06:43:00.625ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19742025-04-30T06:43:00.625ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19752025-04-30T06:43:00.625ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19762025-04-30T06:43:00.626ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19772025-04-30T06:43:00.626ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d7 49 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .I..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56404 req_id = c1b3b814-b47c-4bc1-98cd-66a3ea44dd14
19782025-04-30T06:43:00.627ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55113,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56404 req_id = c1b3b814-b47c-4bc1-98cd-66a3ea44dd14
19792025-04-30T06:43:00.627ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
19802025-04-30T06:43:00.628ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19812025-04-30T06:43:00.628ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56404 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c1b3b814-b47c-4bc1-98cd-66a3ea44dd14
19822025-04-30T06:43:00.628ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4245, tv_nsec: 427613412 } })
19832025-04-30T06:43:00.628ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
19842025-04-30T06:43:00.634ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19852025-04-30T06:43:00.634ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
19862025-04-30T06:43:00.635ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19872025-04-30T06:43:00.636ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19882025-04-30T06:43:00.650ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19892025-04-30T06:43:00.650ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19902025-04-30T06:43:00.651ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19912025-04-30T06:43:00.651ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19922025-04-30T06:43:00.652ZDEBGtest_snapshot_unwind (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19932025-04-30T06:43:00.652ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19942025-04-30T06:43:00.652ZDEBGtest_snapshot_unwind (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19952025-04-30T06:43:00.652ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19962025-04-30T06:43:00.652ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19972025-04-30T06:43:00.652ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19982025-04-30T06:43:00.652ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19992025-04-30T06:43:00.653ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20002025-04-30T06:43:00.654ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e1 7b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .{..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49521 req_id = f974a5b3-7aff-444c-a121-1b9de7a39d69
20012025-04-30T06:43:00.654ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57723,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49521 req_id = f974a5b3-7aff-444c-a121-1b9de7a39d69
20022025-04-30T06:43:00.654ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20032025-04-30T06:43:00.655ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49521 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f974a5b3-7aff-444c-a121-1b9de7a39d69
20042025-04-30T06:43:00.734ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20052025-04-30T06:43:00.735ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20062025-04-30T06:43:00.740ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20072025-04-30T06:43:00.741ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20082025-04-30T06:43:00.741ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20092025-04-30T06:43:00.741ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20102025-04-30T06:43:00.741ZDEBGtest_snapshot_unwind (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20112025-04-30T06:43:00.741ZDEBGtest_snapshot_unwind (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7a2baea3-141c-4d7d-9578-949c38f3fe1f dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20122025-04-30T06:43:00.741ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20132025-04-30T06:43:00.741ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20142025-04-30T06:43:00.741ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20152025-04-30T06:43:00.741ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20162025-04-30T06:43:00.742ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20172025-04-30T06:43:00.742ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20182025-04-30T06:43:00.742ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20192025-04-30T06:43:00.742ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20202025-04-30T06:43:00.742ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20212025-04-30T06:43:00.743ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20222025-04-30T06:43:00.745ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20232025-04-30T06:43:00.745ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
20242025-04-30T06:43:00.745ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20252025-04-30T06:43:00.746ZDEBGtest_snapshot_unwind (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20262025-04-30T06:43:00.746ZDEBGtest_snapshot_unwind (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e1418867-6651-4fd0-98d2-f71aed3ecd4d dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20272025-04-30T06:43:00.746ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20282025-04-30T06:43:00.746ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20292025-04-30T06:43:00.746ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20302025-04-30T06:43:00.746ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20312025-04-30T06:43:00.746ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20322025-04-30T06:43:00.747ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f3 33 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .3..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52745 req_id = ee54d489-c0fb-4094-9ea2-cdc6039d2a81
20332025-04-30T06:43:00.747ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62259,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52745 req_id = ee54d489-c0fb-4094-9ea2-cdc6039d2a81
20342025-04-30T06:43:00.747ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
20352025-04-30T06:43:00.747ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62259, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52745 req_id = ee54d489-c0fb-4094-9ea2-cdc6039d2a81
20362025-04-30T06:43:00.747ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
20372025-04-30T06:43:00.747ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
20382025-04-30T06:43:00.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
20392025-04-30T06:43:00.749ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20402025-04-30T06:43:00.759ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20412025-04-30T06:43:00.759ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20422025-04-30T06:43:00.759ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20432025-04-30T06:43:00.759ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20442025-04-30T06:43:00.760ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20452025-04-30T06:43:00.764ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20462025-04-30T06:43:00.764ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20472025-04-30T06:43:00.764ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20482025-04-30T06:43:00.766ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20492025-04-30T06:43:00.768ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20502025-04-30T06:43:00.776ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20512025-04-30T06:43:00.776ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20522025-04-30T06:43:00.776ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20532025-04-30T06:43:00.776ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20542025-04-30T06:43:00.779ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20552025-04-30T06:43:00.779ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20562025-04-30T06:43:00.779ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20572025-04-30T06:43:00.779ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
20582025-04-30T06:43:00.780ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20592025-04-30T06:43:00.785ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e299ad-50d2-4a0a-b383-99fb658e89a1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
20602025-04-30T06:43:00.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
20612025-04-30T06:43:00.786ZDEBGtest_snapshot_unwind (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e1418867-6651-4fd0-98d2-f71aed3ecd4d dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20622025-04-30T06:43:00.786ZDEBGtest_snapshot_unwind (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20632025-04-30T06:43:00.786ZINFOtest_snapshot_unwind (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20642025-04-30T06:43:00.786ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 272.954172ms iteration = 2 status = {"generation":2}
20652025-04-30T06:43:00.786ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20662025-04-30T06:43:00.871ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:40326 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:40326] } uri = http://[::1]:40326/config
20672025-04-30T06:43:00.872ZDEBGtest_snapshot_unwind (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7a2baea3-141c-4d7d-9578-949c38f3fe1f dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20682025-04-30T06:43:00.872ZDEBGtest_snapshot_unwind (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20692025-04-30T06:43:00.872ZINFOtest_snapshot_unwind (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20702025-04-30T06:43:00.872ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 360.046087ms iteration = 2 status = {"generation":1}
20712025-04-30T06:43:00.872ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20722025-04-30T06:43:00.937ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:54858 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:54858] } uri = http://[::1]:54858/config
20732025-04-30T06:43:00.938ZINFOtest_snapshot_unwind (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:42:52.497349Z, time_modified: 2025-04-30T06:42:52.497349Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 9587148f-d9a3-4861-a834-bcd2b615e746, db_silo: Silo { identity: SiloIdentity { id: 9587148f-d9a3-4861-a834-bcd2b615e746, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:59.659984Z, time_modified: 2025-04-30T06:42:59.659984Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 9587148f-d9a3-4861-a834-bcd2b615e746, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 9587148f-d9a3-4861-a834-bcd2b615e746, db_silo: Silo { identity: SiloIdentity { id: 9587148f-d9a3-4861-a834-bcd2b615e746, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:42:59.659984Z, time_modified: 2025-04-30T06:42:59.659984Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20742025-04-30T06:43:00.938ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 423.409285ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"9587148f-d9a3-4861-a834-bcd2b615e746","tls_certs":[]}},"default_endpoint":{"silo_id":"9587148f-d9a3-4861-a834-bcd2b615e746","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 9587148f-d9a3-4861-a834-bcd2b615e746 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20752025-04-30T06:43:00.939ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20762025-04-30T06:43:00.939ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20772025-04-30T06:43:00.940ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
20782025-04-30T06:43:00.941ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20792025-04-30T06:43:00.941ZINFOtest_snapshot_unwind (http): accepted connection kind = dns local_addr = [::1]:54858 remote_addr = [::1]:58791
20802025-04-30T06:43:00.941ZINFOtest_snapshot_unwind (http): accepted connection kind = dns local_addr = [::1]:40326 remote_addr = [::1]:35824
20812025-04-30T06:43:00.945ZTRACtest_snapshot_unwind (http): incoming request kind = dns local_addr = [::1]:54858 method = PUT remote_addr = [::1]:58791 req_id = 3efc2911-92be-4f1f-bdb5-fc0f6a09ac9a uri = /config
20822025-04-30T06:43:00.945ZTRACtest_snapshot_unwind (http): incoming request kind = dns local_addr = [::1]:40326 method = PUT remote_addr = [::1]:35824 req_id = 6fcefbeb-4818-4632-9152-2cfa6906222b uri = /config
20832025-04-30T06:43:00.945ZINFOtest_snapshot_unwind (store): attempting generation update new_generation = 1 req_id = 3efc2911-92be-4f1f-bdb5-fc0f6a09ac9a
20842025-04-30T06:43:00.945ZINFOtest_snapshot_unwind (store): updated generation new_generation = 1 req_id = 3efc2911-92be-4f1f-bdb5-fc0f6a09ac9a
20852025-04-30T06:43:00.945ZINFOtest_snapshot_unwind (store): attempting generation update new_generation = 2 req_id = 6fcefbeb-4818-4632-9152-2cfa6906222b
20862025-04-30T06:43:00.945ZINFOtest_snapshot_unwind (store): pruning trees for generations newer than 0
20872025-04-30T06:43:00.945ZDEBGtest_snapshot_unwind (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20882025-04-30T06:43:00.945ZDEBGtest_snapshot_unwind (store): updating current config
20892025-04-30T06:43:00.946ZDEBGtest_snapshot_unwind (store): flushing default tree
20902025-04-30T06:43:00.946ZINFOtest_snapshot_unwind (store): pruning trees for generations older than 2 keep = 3
20912025-04-30T06:43:00.946ZINFOtest_snapshot_unwind (store): updated generation new_generation = 2 req_id = 6fcefbeb-4818-4632-9152-2cfa6906222b
20922025-04-30T06:43:00.946ZINFOtest_snapshot_unwind (http): request completed kind = dns latency_us = 1884 local_addr = [::1]:54858 method = PUT remote_addr = [::1]:58791 req_id = 3efc2911-92be-4f1f-bdb5-fc0f6a09ac9a response_code = 204 uri = /config
20932025-04-30T06:43:00.946ZINFOtest_snapshot_unwind (http): request completed kind = dns latency_us = 1820 local_addr = [::1]:40326 method = PUT remote_addr = [::1]:35824 req_id = 6fcefbeb-4818-4632-9152-2cfa6906222b response_code = 204 uri = /config
20942025-04-30T06:43:00.946ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20952025-04-30T06:43:00.946ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20962025-04-30T06:43:00.946ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
20972025-04-30T06:43:00.947ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20982025-04-30T06:43:00.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:40326 generation = 2 result = Ok(Response { url: "http://[::1]:40326/config", status: 204, headers: {"x-request-id": "6fcefbeb-4818-4632-9152-2cfa6906222b", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} }) servers = DnsServersList { addresses: [[::1]:40326] }
20992025-04-30T06:43:00.948ZINFOtest_snapshot_unwind (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:40326 generation = 2 servers = DnsServersList { addresses: [[::1]:40326] }
21002025-04-30T06:43:00.950ZINFOtest_snapshot_unwind (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:40326] }
21012025-04-30T06:43:00.951ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 164.638001ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:40326":{"Ok":null}}}
21022025-04-30T06:43:00.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:54858 generation = 1 result = Ok(Response { url: "http://[::1]:54858/config", status: 204, headers: {"x-request-id": "3efc2911-92be-4f1f-bdb5-fc0f6a09ac9a", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} }) servers = DnsServersList { addresses: [[::1]:54858] }
21032025-04-30T06:43:00.951ZINFOtest_snapshot_unwind (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:54858 generation = 1 servers = DnsServersList { addresses: [[::1]:54858] }
21042025-04-30T06:43:00.954ZINFOtest_snapshot_unwind (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:54858] }
21052025-04-30T06:43:00.954ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 81.938648ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:54858":{"Ok":null}}}
21062025-04-30T06:43:00.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9a6040-f352-4515-b80d-be7adf8c64ea", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21072025-04-30T06:43:00.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
21082025-04-30T06:43:00.958ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-04-30T06:43:00.968ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21102025-04-30T06:43:00.968ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21112025-04-30T06:43:00.968ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21122025-04-30T06:43:00.969ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21132025-04-30T06:43:00.969ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21142025-04-30T06:43:00.969ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21152025-04-30T06:43:00.969ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21162025-04-30T06:43:00.972ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1afe7231-8ef0-44f1-a5f5-a0d5cb81184d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21172025-04-30T06:43:00.972ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
21182025-04-30T06:43:00.973ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21192025-04-30T06:43:00.973ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21202025-04-30T06:43:00.973ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21212025-04-30T06:43:00.974ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21222025-04-30T06:43:00.976ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21232025-04-30T06:43:00.977ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21242025-04-30T06:43:00.977ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21252025-04-30T06:43:00.977ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21262025-04-30T06:43:00.977ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d53c0e2c-c32d-42b9-9cfc-9423db3feabf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21272025-04-30T06:43:00.977ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
21282025-04-30T06:43:00.980ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21292025-04-30T06:43:00.989ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21302025-04-30T06:43:00.989ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21312025-04-30T06:43:00.989ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21322025-04-30T06:43:00.990ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21332025-04-30T06:43:00.991ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f804795-bd84-4582-863d-6f3ad4e284aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21342025-04-30T06:43:00.991ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
21352025-04-30T06:43:00.994ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21362025-04-30T06:43:00.994ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21372025-04-30T06:43:00.996ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21382025-04-30T06:43:01.001ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:43:01.001ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:43:01.002ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:43:01.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7a5fab-a2a0-4cc5-9b84-cd28d8146806", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21422025-04-30T06:43:01.006ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
21432025-04-30T06:43:01.006ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21442025-04-30T06:43:01.006ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21452025-04-30T06:43:01.008ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21462025-04-30T06:43:01.012ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21472025-04-30T06:43:01.012ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21482025-04-30T06:43:01.013ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21492025-04-30T06:43:01.013ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21502025-04-30T06:43:01.013ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21512025-04-30T06:43:01.014ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c40a40b6-97ca-489e-bdb0-7e6a4d8ffc98", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21522025-04-30T06:43:01.014ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
21532025-04-30T06:43:01.017ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21542025-04-30T06:43:01.017ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21552025-04-30T06:43:01.017ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21562025-04-30T06:43:01.018ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21572025-04-30T06:43:01.021ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5eee933e-2d0a-4dbf-862a-2bd352c15825", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21582025-04-30T06:43:01.021ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
21592025-04-30T06:43:01.021ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21602025-04-30T06:43:01.021ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21612025-04-30T06:43:01.021ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21622025-04-30T06:43:01.022ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21632025-04-30T06:43:01.023ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd00bc15-ec64-4983-b965-dc066c71b980", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21642025-04-30T06:43:01.023ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
21652025-04-30T06:43:01.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bddb1938-4977-4d39-8c47-dbeb24dc0763", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21662025-04-30T06:43:01.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
21672025-04-30T06:43:01.026ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21682025-04-30T06:43:01.038ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21692025-04-30T06:43:01.039ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21702025-04-30T06:43:01.039ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21712025-04-30T06:43:01.039ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21722025-04-30T06:43:01.040ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21732025-04-30T06:43:01.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b0ea7bc-5605-4bd9-a0aa-4ebb6236b5d3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21742025-04-30T06:43:01.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
21752025-04-30T06:43:01.044ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21762025-04-30T06:43:01.045ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21772025-04-30T06:43:01.046ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21782025-04-30T06:43:01.046ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03d34b2d-ef94-4bbc-aff8-53d99398afad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21792025-04-30T06:43:01.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
21802025-04-30T06:43:01.049ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a28f2d0-098b-4ada-bcfe-b12028df6c81", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21812025-04-30T06:43:01.049ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
21822025-04-30T06:43:01.050ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21832025-04-30T06:43:01.050ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21842025-04-30T06:43:01.050ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21852025-04-30T06:43:01.054ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d9de99-b256-4b6a-a019-df1560f44ae2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21862025-04-30T06:43:01.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
21872025-04-30T06:43:01.055ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21882025-04-30T06:43:01.055ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21892025-04-30T06:43:01.056ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21902025-04-30T06:43:01.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc56c439-cd4b-459f-9712-fe1e7c5c0ed1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
21912025-04-30T06:43:01.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
21922025-04-30T06:43:01.060ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
21932025-04-30T06:43:01.060ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21942025-04-30T06:43:01.060ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21952025-04-30T06:43:01.060ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
21962025-04-30T06:43:01.061ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21972025-04-30T06:43:01.064ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21982025-04-30T06:43:01.064ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21992025-04-30T06:43:01.064ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22002025-04-30T06:43:01.065ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22012025-04-30T06:43:01.065ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0216e4a3-c770-4c1a-93c8-da07004405a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22022025-04-30T06:43:01.065ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
22032025-04-30T06:43:01.068ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "257f365f-89e5-4c6e-ae64-f5f09b8e6435", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22042025-04-30T06:43:01.068ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
22052025-04-30T06:43:01.069ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22062025-04-30T06:43:01.069ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22072025-04-30T06:43:01.069ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22082025-04-30T06:43:01.070ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22092025-04-30T06:43:01.074ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22102025-04-30T06:43:01.082ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22112025-04-30T06:43:01.083ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22122025-04-30T06:43:01.083ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22132025-04-30T06:43:01.083ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22142025-04-30T06:43:01.083ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22152025-04-30T06:43:01.083ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22162025-04-30T06:43:01.083ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22172025-04-30T06:43:01.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "614615fa-92bb-43e0-89d8-7732b10b696b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22182025-04-30T06:43:01.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
22192025-04-30T06:43:01.087ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22202025-04-30T06:43:01.087ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22212025-04-30T06:43:01.087ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22222025-04-30T06:43:01.088ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22232025-04-30T06:43:01.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9250c384-6410-40b2-9864-9a3529792a8f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22242025-04-30T06:43:01.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
22252025-04-30T06:43:01.090ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22262025-04-30T06:43:01.091ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22272025-04-30T06:43:01.094ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22282025-04-30T06:43:01.094ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22292025-04-30T06:43:01.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "926d7838-3613-4500-9447-8dfc9d8b149f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22302025-04-30T06:43:01.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
22312025-04-30T06:43:01.094ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22322025-04-30T06:43:01.105ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22332025-04-30T06:43:01.105ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22342025-04-30T06:43:01.105ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22352025-04-30T06:43:01.107ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22362025-04-30T06:43:01.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c068327-60fd-4c5a-a204-662c48e0a73a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22372025-04-30T06:43:01.109ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
22382025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad8d0c43-458f-4848-bbec-9ecfe8255e7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22392025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
22402025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa0401d-b3db-42d2-9195-2756923b5715", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22412025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
22422025-04-30T06:43:01.113ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22432025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22442025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22452025-04-30T06:43:01.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee582fd0-8698-47c8-b0fc-71a69aaa2ac1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22462025-04-30T06:43:01.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
22472025-04-30T06:43:01.124ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22482025-04-30T06:43:01.124ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22492025-04-30T06:43:01.125ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22502025-04-30T06:43:01.128ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ee34e3-b08e-48ed-8a0d-a1edb5975b4c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22512025-04-30T06:43:01.128ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
22522025-04-30T06:43:01.128ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22532025-04-30T06:43:01.128ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22542025-04-30T06:43:01.130ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22552025-04-30T06:43:01.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b66f9b7-272d-4a4a-af05-b76357ebd724", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22562025-04-30T06:43:01.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
22572025-04-30T06:43:01.133ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22582025-04-30T06:43:01.134ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22592025-04-30T06:43:01.134ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22602025-04-30T06:43:01.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3e72a5-2eb7-433d-923e-697333664173", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22612025-04-30T06:43:01.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
22622025-04-30T06:43:01.137ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22632025-04-30T06:43:01.138ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22642025-04-30T06:43:01.138ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22652025-04-30T06:43:01.138ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22662025-04-30T06:43:01.138ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22672025-04-30T06:43:01.141ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22682025-04-30T06:43:01.141ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22692025-04-30T06:43:01.142ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22702025-04-30T06:43:01.143ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22712025-04-30T06:43:01.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9845172d-c7ae-4910-b962-00787bc80109", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22722025-04-30T06:43:01.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
22732025-04-30T06:43:01.146ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22742025-04-30T06:43:01.146ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22752025-04-30T06:43:01.146ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22762025-04-30T06:43:01.147ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22772025-04-30T06:43:01.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bde907c9-c7bb-46ca-bff2-0dae2e74ee44", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22782025-04-30T06:43:01.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
22792025-04-30T06:43:01.150ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22802025-04-30T06:43:01.167ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22812025-04-30T06:43:01.168ZTRACtest_snapshot_unwind (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22822025-04-30T06:43:01.168ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22832025-04-30T06:43:01.168ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22842025-04-30T06:43:01.168ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22852025-04-30T06:43:01.168ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22862025-04-30T06:43:01.168ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22872025-04-30T06:43:01.171ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22882025-04-30T06:43:01.172ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22892025-04-30T06:43:01.172ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22902025-04-30T06:43:01.172ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22912025-04-30T06:43:01.175ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22922025-04-30T06:43:01.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22932025-04-30T06:43:01.175ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
22942025-04-30T06:43:01.175ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22952025-04-30T06:43:01.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11c2e7be-a0ac-4960-9f90-df1fe983852d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22962025-04-30T06:43:01.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
22972025-04-30T06:43:01.177ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56fcf135-3cb9-4217-b69c-bc0365c3abab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
22982025-04-30T06:43:01.177ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
22992025-04-30T06:43:01.184ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23002025-04-30T06:43:01.195ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
23012025-04-30T06:43:01.195ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23022025-04-30T06:43:01.195ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
23032025-04-30T06:43:01.195ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23042025-04-30T06:43:01.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6e91a31-f490-484c-884f-a4b8b62c3e62", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23052025-04-30T06:43:01.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
23062025-04-30T06:43:01.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbc0bc45-56b9-40d3-94e7-e80b3993af21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23072025-04-30T06:43:01.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
23082025-04-30T06:43:01.206ZDEBGtest_snapshot_unwind (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 03c0792c-5eb3-47a6-ba0a-3459a993795f, kind: Service { id: 6c4c48b5-9647-4b7a-ba06-9cebd7318846 }, name: Name("external-dns-6c4c48b5-9647-4b7a-ba06-9cebd7318846"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 581ba028-07c9-4415-b60d-b5925e41fb0a, kind: Service { id: f7a56cc4-6855-4ee1-bf23-43c5f437915c }, name: Name("nexus-f7a56cc4-6855-4ee1-bf23-43c5f437915c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
23092025-04-30T06:43:01.206ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23102025-04-30T06:43:01.206ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23112025-04-30T06:43:01.206ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23122025-04-30T06:43:01.207ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
23132025-04-30T06:43:01.207ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23142025-04-30T06:43:01.207ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33e8f075-a74b-4ba1-b758-c19a292e8c2b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23152025-04-30T06:43:01.207ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
23162025-04-30T06:43:01.207ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23172025-04-30T06:43:01.224ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23182025-04-30T06:43:01.224ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23192025-04-30T06:43:01.224ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
23202025-04-30T06:43:01.233ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23212025-04-30T06:43:01.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f1779ee-5588-4aca-99a3-df17a9272411", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23222025-04-30T06:43:01.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
23232025-04-30T06:43:01.239ZTRACtest_snapshot_unwind (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:42:59.847845Z, allowed_ips: None }
23242025-04-30T06:43:01.240ZDEBGtest_snapshot_unwind (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23252025-04-30T06:43:01.240ZDEBGtest_snapshot_unwind (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 03c0792c-5eb3-47a6-ba0a-3459a993795f, kind: Service { id: 6c4c48b5-9647-4b7a-ba06-9cebd7318846 }, name: Name("external-dns-6c4c48b5-9647-4b7a-ba06-9cebd7318846"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 581ba028-07c9-4415-b60d-b5925e41fb0a, kind: Service { id: f7a56cc4-6855-4ee1-bf23-43c5f437915c }, name: Name("nexus-f7a56cc4-6855-4ee1-bf23-43c5f437915c"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23262025-04-30T06:43:01.240ZDEBGtest_snapshot_unwind (InternalApi): resolved 2 rules for sleds
23272025-04-30T06:43:01.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce1937fe-d6bb-419c-94e9-762e9cf065a8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23282025-04-30T06:43:01.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:43:01.256ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2152a94c-24d7-45bf-a9f1-72d969d9e6b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23302025-04-30T06:43:01.256ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2ce2c4-61bf-41f6-ba81-63956b5ee7d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23322025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1bd809f-c960-4c15-9a74-d27d8469aaeb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23342025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
23352025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3e083a0-5fb8-46b1-9ba0-e5a05f1e9f0b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23362025-04-30T06:43:01.257ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
23372025-04-30T06:43:01.275ZDEBGtest_snapshot_unwind (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:42:54.200591Z, time_modified: 2025-04-30T06:42:54.200591Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(45714), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(63004) }]
23382025-04-30T06:43:01.275ZDEBGtest_snapshot_unwind (InternalApi): sending firewall rules to sled agents
23392025-04-30T06:43:01.275ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23402025-04-30T06:43:01.275ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
23412025-04-30T06:43:01.283ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23422025-04-30T06:43:01.287ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23432025-04-30T06:43:01.287ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23442025-04-30T06:43:01.287ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23452025-04-30T06:43:01.287ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
23462025-04-30T06:43:01.287ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23472025-04-30T06:43:01.288ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23482025-04-30T06:43:01.295ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23492025-04-30T06:43:01.348ZDEBGtest_snapshot_unwind (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:45714/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23502025-04-30T06:43:01.348ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a9ecbc4-e5e0-4cb2-83b5-4ea2699a2bf5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23512025-04-30T06:43:01.349ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
23522025-04-30T06:43:01.349ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:35174 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23532025-04-30T06:43:01.352ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:35174 req_id = 5ae4582d-0e75-42a3-9a02-6f437af6023e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23542025-04-30T06:43:01.355ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1209 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:35174 req_id = 5ae4582d-0e75-42a3-9a02-6f437af6023e response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23552025-04-30T06:43:01.355ZDEBGtest_snapshot_unwind (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:45714/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "5ae4582d-0e75-42a3-9a02-6f437af6023e", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
23562025-04-30T06:43:01.355ZINFOtest_snapshot_unwind (InternalApi): updated firewall rules on 1 sleds
23572025-04-30T06:43:01.356ZINFOtest_snapshot_unwind (nexus): plumbed initial IP allowlist
23582025-04-30T06:43:01.407ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = / versions = all
23592025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /access versions = all
23602025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /assets/{path} versions = all
23612025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /device/auth versions = all
23622025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /device/confirm versions = all
23632025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /device/success versions = all
23642025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /device/token versions = all
23652025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /device/verify versions = all
23662025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/probes versions = all
23672025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /experimental/v1/probes versions = all
23682025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23692025-04-30T06:43:01.408ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/probes/{probe} versions = all
23702025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/system/support-bundles versions = all
23712025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /experimental/v1/system/support-bundles versions = all
23722025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23732025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23742025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23752025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23762025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23772025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23782025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23792025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /images versions = all
23802025-04-30T06:43:01.409ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /login versions = all
23812025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /login/{silo_name}/local versions = all
23822025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23832025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23842025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23852025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /lookup/{path} versions = all
23862025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /projects/{path} versions = all
23872025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /projects-new versions = all
23882025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /settings/{path} versions = all
23892025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /system/{path} versions = all
23902025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /utilization versions = all
23912025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/affinity-groups versions = all
23922025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/affinity-groups versions = all
23932025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
23942025-04-30T06:43:01.410ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
23952025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
23962025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
23972025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23982025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
23992025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24002025-04-30T06:43:01.411ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/anti-affinity-groups versions = all
24012025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/anti-affinity-groups versions = all
24022025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24032025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24042025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24052025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24062025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24072025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24082025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24092025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/certificates versions = all
24102025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/certificates versions = all
24112025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/certificates/{certificate} versions = all
24122025-04-30T06:43:01.412ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/certificates/{certificate} versions = all
24132025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/disks versions = all
24142025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/disks versions = all
24152025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/disks/{disk} versions = all
24162025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/disks/{disk} versions = all
24172025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24182025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24192025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24202025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/disks/{disk}/finalize versions = all
24212025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24222025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/floating-ips versions = all
24232025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/floating-ips versions = all
24242025-04-30T06:43:01.413ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24252025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24262025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24272025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24282025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24292025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/groups versions = all
24302025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/groups/{group_id} versions = all
24312025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/images versions = all
24322025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/images versions = all
24332025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/images/{image} versions = all
24342025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/images/{image} versions = all
24352025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/images/{image}/demote versions = all
24362025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/images/{image}/promote versions = all
24372025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances versions = all
24382025-04-30T06:43:01.414ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances versions = all
24392025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/instances/{instance} versions = all
24402025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance} versions = all
24412025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/instances/{instance} versions = all
24422025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24432025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24442025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/disks versions = all
24452025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24462025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24472025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/external-ips versions = all
24482025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24492025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24502025-04-30T06:43:01.415ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/reboot versions = all
24512025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/serial-console versions = all
24522025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24532025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24542025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/start versions = all
24552025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/instances/{instance}/stop versions = all
24562025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24572025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24582025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24592025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/internet-gateway-ip-pools versions = all
24602025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/internet-gateway-ip-pools versions = all
24612025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24622025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/internet-gateways versions = all
24632025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/internet-gateways versions = all
24642025-04-30T06:43:01.416ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24652025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/internet-gateways/{gateway} versions = all
24662025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/ip-pools versions = all
24672025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/ip-pools/{pool} versions = all
24682025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/login/{silo_name}/local versions = all
24692025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/logout versions = all
24702025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/me versions = all
24712025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/me/groups versions = all
24722025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/me/ssh-keys versions = all
24732025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/me/ssh-keys versions = all
24742025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24752025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24762025-04-30T06:43:01.417ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/metrics/{metric_name} versions = all
24772025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/network-interfaces versions = all
24782025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/network-interfaces versions = all
24792025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24802025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/network-interfaces/{interface} versions = all
24812025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/network-interfaces/{interface} versions = all
24822025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/ping versions = all
24832025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/policy versions = all
24842025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/policy versions = all
24852025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/projects versions = all
24862025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/projects versions = all
24872025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/projects/{project} versions = all
24882025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/projects/{project} versions = all
24892025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/projects/{project} versions = all
24902025-04-30T06:43:01.418ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/projects/{project}/policy versions = all
24912025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/projects/{project}/policy versions = all
24922025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/snapshots versions = all
24932025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/snapshots versions = all
24942025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/snapshots/{snapshot} versions = all
24952025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/snapshots/{snapshot} versions = all
24962025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/disks versions = all
24972025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
24982025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
24992025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/racks versions = all
25002025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25012025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/sleds versions = all
25022025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/hardware/sleds versions = all
25032025-04-30T06:43:01.419ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25042025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25052025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25062025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25072025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25082025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/switch-port versions = all
25092025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25102025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25112025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25122025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25132025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25142025-04-30T06:43:01.420ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/switches versions = all
25152025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25162025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/identity-providers versions = all
25172025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/identity-providers/local/users versions = all
25182025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25192025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25202025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/identity-providers/saml versions = all
25212025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25222025-04-30T06:43:01.421ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools versions = all
25232025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools versions = all
25242025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25252025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools/{pool} versions = all
25262025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25272025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25282025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25292025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25302025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25312025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25322025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25332025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25342025-04-30T06:43:01.422ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25352025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools-service versions = all
25362025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25372025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25382025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25392025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/metrics/{metric_name} versions = all
25402025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/address-lot versions = all
25412025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/address-lot versions = all
25422025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25432025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25442025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/allow-list versions = all
25452025-04-30T06:43:01.423ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/networking/allow-list versions = all
25462025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/bfd-disable versions = all
25472025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/bfd-enable versions = all
25482025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bfd-status versions = all
25492025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/networking/bgp versions = all
25502025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp versions = all
25512025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/bgp versions = all
25522025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25532025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25542025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25552025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25562025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-exported versions = all
25572025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-message-history versions = all
25582025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25592025-04-30T06:43:01.424ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/bgp-status versions = all
25602025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/loopback-address versions = all
25612025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/loopback-address versions = all
25622025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25632025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25642025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/switch-port-settings versions = all
25652025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/networking/switch-port-settings versions = all
25662025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25672025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/policy versions = all
25682025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/policy versions = all
25692025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/roles versions = all
25702025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/roles/{role_name} versions = all
25712025-04-30T06:43:01.425ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silo-quotas versions = all
25722025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silos versions = all
25732025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/silos versions = all
25742025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/system/silos/{silo} versions = all
25752025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silos/{silo} versions = all
25762025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25772025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silos/{silo}/policy versions = all
25782025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25792025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25802025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25812025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/system/timeseries/query versions = all
25822025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/timeseries/schemas versions = all
25832025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/update/repository versions = all
25842025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/update/repository/{system_version} versions = all
25852025-04-30T06:43:01.426ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/update/target-release versions = all
25862025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/system/update/target-release versions = all
25872025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/users versions = all
25882025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/users/{user_id} versions = all
25892025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/users-builtin versions = all
25902025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/users-builtin/{user} versions = all
25912025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/utilization/silos versions = all
25922025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/system/utilization/silos/{silo} versions = all
25932025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/timeseries/query versions = all
25942025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/users versions = all
25952025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/utilization versions = all
25962025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-firewall-rules versions = all
25972025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/vpc-firewall-rules versions = all
25982025-04-30T06:43:01.427ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-router-routes versions = all
25992025-04-30T06:43:01.428ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/vpc-router-routes versions = all
26002025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26012025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-router-routes/{route} versions = all
26022025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26032025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-routers versions = all
26042025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/vpc-routers versions = all
26052025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/vpc-routers/{router} versions = all
26062025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-routers/{router} versions = all
26072025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/vpc-routers/{router} versions = all
26082025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-subnets versions = all
26092025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/vpc-subnets versions = all
26102025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26112025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26122025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26132025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26142025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpcs versions = all
26152025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/vpcs versions = all
26162025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/vpcs/{vpc} versions = all
26172025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/vpcs/{vpc} versions = all
26182025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/vpcs/{vpc} versions = all
26192025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/webhooks/deliveries versions = all
26202025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26212025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/webhooks/event-classes versions = all
26222025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/webhooks/receivers versions = all
26232025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/webhooks/receivers versions = all
26242025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26252025-04-30T06:43:01.430ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26262025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26272025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26282025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26292025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26302025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = GET path = /v1/webhooks/secrets versions = all
26312025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = POST path = /v1/webhooks/secrets versions = all
26322025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): registered endpoint local_addr = 127.0.0.1:45927 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26332025-04-30T06:43:01.431ZINFOtest_snapshot_unwind (dropshot_external): listening local_addr = 127.0.0.1:45927
26342025-04-30T06:43:01.431ZDEBGtest_snapshot_unwind (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:45927
26352025-04-30T06:43:01.489ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = / versions = all
26362025-04-30T06:43:01.489ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /access versions = all
26372025-04-30T06:43:01.489ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /assets/{path} versions = all
26382025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /device/auth versions = all
26392025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /device/confirm versions = all
26402025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /device/success versions = all
26412025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /device/token versions = all
26422025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /device/verify versions = all
26432025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/probes versions = all
26442025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /experimental/v1/probes versions = all
26452025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26462025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/probes/{probe} versions = all
26472025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/system/support-bundles versions = all
26482025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /experimental/v1/system/support-bundles versions = all
26492025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26502025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26512025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26522025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26532025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26542025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26552025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26562025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /images versions = all
26572025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /login versions = all
26582025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /login/{silo_name}/local versions = all
26592025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26602025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26612025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26622025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /lookup/{path} versions = all
26632025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /projects/{path} versions = all
26642025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /projects-new versions = all
26652025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /settings/{path} versions = all
26662025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /system/{path} versions = all
26672025-04-30T06:43:01.490ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /utilization versions = all
26682025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/affinity-groups versions = all
26692025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/affinity-groups versions = all
26702025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26712025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26722025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26732025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26742025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26752025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26762025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26772025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/anti-affinity-groups versions = all
26782025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/anti-affinity-groups versions = all
26792025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26802025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26812025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26822025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26832025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26842025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26852025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26862025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/certificates versions = all
26872025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/certificates versions = all
26882025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/certificates/{certificate} versions = all
26892025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/certificates/{certificate} versions = all
26902025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/disks versions = all
26912025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/disks versions = all
26922025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/disks/{disk} versions = all
26932025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/disks/{disk} versions = all
26942025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/disks/{disk}/bulk-write versions = all
26952025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
26962025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
26972025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/disks/{disk}/finalize versions = all
26982025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
26992025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/floating-ips versions = all
27002025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/floating-ips versions = all
27012025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27022025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27032025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27042025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27052025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27062025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/groups versions = all
27072025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/groups/{group_id} versions = all
27082025-04-30T06:43:01.491ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/images versions = all
27092025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/images versions = all
27102025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/images/{image} versions = all
27112025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/images/{image} versions = all
27122025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/images/{image}/demote versions = all
27132025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/images/{image}/promote versions = all
27142025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances versions = all
27152025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances versions = all
27162025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/instances/{instance} versions = all
27172025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance} versions = all
27182025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/instances/{instance} versions = all
27192025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27202025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27212025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/disks versions = all
27222025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27232025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27242025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/external-ips versions = all
27252025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27262025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27272025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/reboot versions = all
27282025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/serial-console versions = all
27292025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27302025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27312025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/start versions = all
27322025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/instances/{instance}/stop versions = all
27332025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27342025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27352025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27362025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/internet-gateway-ip-pools versions = all
27372025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/internet-gateway-ip-pools versions = all
27382025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27392025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/internet-gateways versions = all
27402025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/internet-gateways versions = all
27412025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27422025-04-30T06:43:01.492ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/internet-gateways/{gateway} versions = all
27432025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/ip-pools versions = all
27442025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/ip-pools/{pool} versions = all
27452025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/login/{silo_name}/local versions = all
27462025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/logout versions = all
27472025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/me versions = all
27482025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/me/groups versions = all
27492025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/me/ssh-keys versions = all
27502025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/me/ssh-keys versions = all
27512025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27522025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27532025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/metrics/{metric_name} versions = all
27542025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/network-interfaces versions = all
27552025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/network-interfaces versions = all
27562025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27572025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/network-interfaces/{interface} versions = all
27582025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/network-interfaces/{interface} versions = all
27592025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/ping versions = all
27602025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/policy versions = all
27612025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/policy versions = all
27622025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/projects versions = all
27632025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/projects versions = all
27642025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/projects/{project} versions = all
27652025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/projects/{project} versions = all
27662025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/projects/{project} versions = all
27672025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/projects/{project}/policy versions = all
27682025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/projects/{project}/policy versions = all
27692025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/snapshots versions = all
27702025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/snapshots versions = all
27712025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27722025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/snapshots/{snapshot} versions = all
27732025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/disks versions = all
27742025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27752025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27762025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/racks versions = all
27772025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27782025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/sleds versions = all
27792025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/hardware/sleds versions = all
27802025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27812025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27822025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27832025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27842025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27852025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/switch-port versions = all
27862025-04-30T06:43:01.493ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27872025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27882025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27892025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27902025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27912025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/switches versions = all
27922025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27932025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/identity-providers versions = all
27942025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/identity-providers/local/users versions = all
27952025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
27962025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
27972025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/identity-providers/saml versions = all
27982025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
27992025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools versions = all
28002025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools versions = all
28012025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28022025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools/{pool} versions = all
28032025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28042025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28052025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28062025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28072025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28082025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28092025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28102025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28112025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28122025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools-service versions = all
28132025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28142025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28152025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28162025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/metrics/{metric_name} versions = all
28172025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/address-lot versions = all
28182025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/address-lot versions = all
28192025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28202025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28212025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/allow-list versions = all
28222025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/networking/allow-list versions = all
28232025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/bfd-disable versions = all
28242025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/bfd-enable versions = all
28252025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bfd-status versions = all
28262025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/networking/bgp versions = all
28272025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp versions = all
28282025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/bgp versions = all
28292025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28302025-04-30T06:43:01.494ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28312025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28322025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28332025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-exported versions = all
28342025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-message-history versions = all
28352025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28362025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/bgp-status versions = all
28372025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/loopback-address versions = all
28382025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/loopback-address versions = all
28392025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28402025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28412025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/switch-port-settings versions = all
28422025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/networking/switch-port-settings versions = all
28432025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28442025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/policy versions = all
28452025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/policy versions = all
28462025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/roles versions = all
28472025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/roles/{role_name} versions = all
28482025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silo-quotas versions = all
28492025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silos versions = all
28502025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/silos versions = all
28512025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/system/silos/{silo} versions = all
28522025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silos/{silo} versions = all
28532025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28542025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silos/{silo}/policy versions = all
28552025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28562025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28572025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28582025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/system/timeseries/query versions = all
28592025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/timeseries/schemas versions = all
28602025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/update/repository versions = all
28612025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/update/repository/{system_version} versions = all
28622025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/update/target-release versions = all
28632025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/system/update/target-release versions = all
28642025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/users versions = all
28652025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/users/{user_id} versions = all
28662025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/users-builtin versions = all
28672025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/users-builtin/{user} versions = all
28682025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/utilization/silos versions = all
28692025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28702025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/timeseries/query versions = all
28712025-04-30T06:43:01.495ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/users versions = all
28722025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/utilization versions = all
28732025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-firewall-rules versions = all
28742025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/vpc-firewall-rules versions = all
28752025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-router-routes versions = all
28762025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/vpc-router-routes versions = all
28772025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28782025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-router-routes/{route} versions = all
28792025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28802025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-routers versions = all
28812025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/vpc-routers versions = all
28822025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/vpc-routers/{router} versions = all
28832025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-routers/{router} versions = all
28842025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/vpc-routers/{router} versions = all
28852025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-subnets versions = all
28862025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/vpc-subnets versions = all
28872025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28882025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28892025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28902025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28912025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpcs versions = all
28922025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/vpcs versions = all
28932025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/vpcs/{vpc} versions = all
28942025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/vpcs/{vpc} versions = all
28952025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/vpcs/{vpc} versions = all
28962025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/webhooks/deliveries versions = all
28972025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
28982025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/webhooks/event-classes versions = all
28992025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/webhooks/receivers versions = all
29002025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/webhooks/receivers versions = all
29012025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29022025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29032025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29042025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29052025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29062025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29072025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = GET path = /v1/webhooks/secrets versions = all
29082025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = POST path = /v1/webhooks/secrets versions = all
29092025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:63949 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29102025-04-30T06:43:01.496ZINFOtest_snapshot_unwind (dropshot_external_techport): listening local_addr = 127.0.0.1:63949
29112025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:63949
29122025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (nexus-producer-server): registered DTrace probes
29132025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = 127.0.0.1:49133 method = GET path = /{producer_id} versions = all
29142025-04-30T06:43:01.496ZINFOtest_snapshot_unwind (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:49133
29152025-04-30T06:43:01.496ZDEBGtest_snapshot_unwind (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:49133
29162025-04-30T06:43:01.497ZDEBGtest_snapshot_unwind (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:49133
29172025-04-30T06:43:01.497ZDEBGtest_snapshot_unwind (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:53046
29182025-04-30T06:43:01.497ZDEBGtest_snapshot_unwind (nexus-producer-server): starting producer registration task
29192025-04-30T06:43:01.497ZINFOtest_snapshot_unwind (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:49133 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
29202025-04-30T06:43:01.497ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_oximeter
29212025-04-30T06:43:01.497ZDEBGtest_snapshot_unwind: Starting Oximeter
29222025-04-30T06:43:01.505ZDEBGtest_snapshot_unwind (oximeter): registered DTrace probes
29232025-04-30T06:43:01.505ZINFOtest_snapshot_unwind (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29242025-04-30T06:43:01.505ZDEBGtest_snapshot_unwind (oximeter): creating ClickHouse client
29252025-04-30T06:43:01.506ZDEBGtest_snapshot_unwind (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29262025-04-30T06:43:01.507ZDEBGtest_snapshot_unwind (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29272025-04-30T06:43:01.507ZDEBGtest_snapshot_unwind (producer-registration-task): will register with Nexus at 127.0.0.1:53046
29282025-04-30T06:43:01.573ZDEBGtest_snapshot_unwind (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:53046/metrics/producers
29292025-04-30T06:43:01.575ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:62237
29302025-04-30T06:43:01.576ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d744158-b539-415b-b7d9-6764287e8cb0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29312025-04-30T06:43:01.576ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
29322025-04-30T06:43:01.579ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:62237 req_id = f0ef1abc-ecfe-4fd7-8009-436c51e98966 uri = /metrics/producers
29332025-04-30T06:43:01.592ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:62237 req_id = f0ef1abc-ecfe-4fd7-8009-436c51e98966 resource = Database uri = /metrics/producers
29342025-04-30T06:43:01.592ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:62237 req_id = f0ef1abc-ecfe-4fd7-8009-436c51e98966 roles = RoleSet { roles: {} } uri = /metrics/producers
29352025-04-30T06:43:01.592ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:62237 req_id = f0ef1abc-ecfe-4fd7-8009-436c51e98966 resource = Database result = Ok(()) uri = /metrics/producers
29362025-04-30T06:43:01.595ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29372025-04-30T06:43:01.604ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e49c183-134c-4e6c-854c-44097b0798e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29382025-04-30T06:43:01.605ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
29392025-04-30T06:43:01.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f1d666-d238-4257-ad6d-72ab4321e985", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29402025-04-30T06:43:01.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
29412025-04-30T06:43:01.609ZWARNtest_snapshot_unwind (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29422025-04-30T06:43:01.609ZDEBGtest_snapshot_unwind (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29432025-04-30T06:43:01.609ZINFOtest_snapshot_unwind (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29442025-04-30T06:43:01.609ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29452025-04-30T06:43:01.612ZWARNtest_snapshot_unwind (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29462025-04-30T06:43:01.612ZINFOtest_snapshot_unwind (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 version = 0
29472025-04-30T06:43:01.612ZINFOtest_snapshot_unwind (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29482025-04-30T06:43:01.612ZDEBGtest_snapshot_unwind (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
29492025-04-30T06:43:01.613ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffe596b-d996-40c9-8e17-8c4e74547a0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29502025-04-30T06:43:01.613ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29512025-04-30T06:43:01.613ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29522025-04-30T06:43:01.613ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29532025-04-30T06:43:01.613ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29542025-04-30T06:43:01.615ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29552025-04-30T06:43:01.616ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = CREATE DATABASE IF NOT EXISTS oximeter
29562025-04-30T06:43:01.617ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29572025-04-30T06:43:01.618ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29582025-04-30T06:43:01.627ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29592025-04-30T06:43:01.630ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29602025-04-30T06:43:01.630ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29612025-04-30T06:43:01.630ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29622025-04-30T06:43:01.630ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29632025-04-30T06:43:01.630ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29642025-04-30T06:43:01.631ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29652025-04-30T06:43:01.635ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29662025-04-30T06:43:01.636ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29672025-04-30T06:43:01.645ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29682025-04-30T06:43:01.645ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29692025-04-30T06:43:01.646ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29702025-04-30T06:43:01.646ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29712025-04-30T06:43:01.647ZINFOtest_snapshot_unwind (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 68084 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:62237 req_id = f0ef1abc-ecfe-4fd7-8009-436c51e98966 response_code = 503 uri = /metrics/producers
29722025-04-30T06:43:01.653ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29732025-04-30T06:43:01.654ZDEBGtest_snapshot_unwind (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "f0ef1abc-ecfe-4fd7-8009-436c51e98966", "content-length": "133", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29742025-04-30T06:43:01.656ZWARNtest_snapshot_unwind (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 148.760706ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"f0ef1abc-ecfe-4fd7-8009-436c51e98966\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:43:00 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"f0ef1abc-ecfe-4fd7-8009-436c51e98966\\" }" file = oximeter/producer/src/lib.rs:371
29752025-04-30T06:43:01.663ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29762025-04-30T06:43:01.674ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29772025-04-30T06:43:01.677ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29782025-04-30T06:43:01.690ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29792025-04-30T06:43:01.759ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
29802025-04-30T06:43:01.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
29812025-04-30T06:43:01.760ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29822025-04-30T06:43:01.763ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:46985 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29832025-04-30T06:43:01.764ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29842025-04-30T06:43:01.765ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:46985 req_id = 82228bb0-01c5-42b9-b8a5-5a8a4e240896 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29852025-04-30T06:43:01.766ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 975 local_addr = [::1]:45714 method = GET remote_addr = [::1]:46985 req_id = 82228bb0-01c5-42b9-b8a5-5a8a4e240896 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29862025-04-30T06:43:01.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "82228bb0-01c5-42b9-b8a5-5a8a4e240896", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:43:00 GMT"} })
29872025-04-30T06:43:01.770ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29882025-04-30T06:43:01.770ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29892025-04-30T06:43:01.770ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
29902025-04-30T06:43:01.774ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
29912025-04-30T06:43:01.774ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29922025-04-30T06:43:01.774ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29932025-04-30T06:43:01.774ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29942025-04-30T06:43:01.775ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29952025-04-30T06:43:01.775ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29962025-04-30T06:43:01.779ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29972025-04-30T06:43:01.783ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
29982025-04-30T06:43:01.783ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29992025-04-30T06:43:01.784ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30002025-04-30T06:43:01.784ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30012025-04-30T06:43:01.785ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30022025-04-30T06:43:01.790ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30032025-04-30T06:43:01.794ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30042025-04-30T06:43:01.799ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30052025-04-30T06:43:01.805ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30062025-04-30T06:43:01.812ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30072025-04-30T06:43:01.825ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30082025-04-30T06:43:01.829ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30092025-04-30T06:43:01.834ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30102025-04-30T06:43:01.841ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30112025-04-30T06:43:01.846ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30122025-04-30T06:43:01.854ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30132025-04-30T06:43:01.861ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30142025-04-30T06:43:01.876ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30152025-04-30T06:43:01.888ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30162025-04-30T06:43:01.911ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30172025-04-30T06:43:01.916ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30182025-04-30T06:43:01.922ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30192025-04-30T06:43:01.925ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30202025-04-30T06:43:01.927ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30212025-04-30T06:43:01.927ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.411377286s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30222025-04-30T06:43:01.928ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30232025-04-30T06:43:01.935ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30242025-04-30T06:43:01.946ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30252025-04-30T06:43:01.953ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30262025-04-30T06:43:01.960ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30272025-04-30T06:43:01.967ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30282025-04-30T06:43:01.974ZDEBGtest_snapshot_unwind (producer-registration-task): will register with Nexus at 127.0.0.1:53046
30292025-04-30T06:43:02.032ZDEBGtest_snapshot_unwind (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:53046/metrics/producers
30302025-04-30T06:43:02.033ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30312025-04-30T06:43:02.033ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:57125
30322025-04-30T06:43:02.033ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:57125 req_id = 6011404e-6afa-4b05-897f-ee04bca0e06e uri = /metrics/producers
30332025-04-30T06:43:02.034ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:57125 req_id = 6011404e-6afa-4b05-897f-ee04bca0e06e resource = Database uri = /metrics/producers
30342025-04-30T06:43:02.034ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:57125 req_id = 6011404e-6afa-4b05-897f-ee04bca0e06e roles = RoleSet { roles: {} } uri = /metrics/producers
30352025-04-30T06:43:02.034ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:57125 req_id = 6011404e-6afa-4b05-897f-ee04bca0e06e resource = Database result = Ok(()) uri = /metrics/producers
30362025-04-30T06:43:02.035ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: b2 02 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:50106 req_id = 578e8dc5-3969-4b1f-91c3-42c10f0e5970
30372025-04-30T06:43:02.035ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45570,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50106 req_id = 578e8dc5-3969-4b1f-91c3-42c10f0e5970
30382025-04-30T06:43:02.036ZDEBGtest_snapshot_unwind (store): query key key = _crucible-pantry._tcp
30392025-04-30T06:43:02.036ZDEBGtest_snapshot_unwind (store): query key key = cef6f1f7-0bb4-4520-9f28-f372c582a644.host
30402025-04-30T06:43:02.036ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50106 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 61186, target: Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal") })) }] req_id = 578e8dc5-3969-4b1f-91c3-42c10f0e5970
30412025-04-30T06:43:02.037ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30422025-04-30T06:43:02.037ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 68 d6 01 00 00 01 00 00 00 00 00 01 24 63 65 66 h...........$cef\n0010: 36 66 31 66 37 2d 30 62 62 34 2d 34 35 32 30 2d 6f1f7-0bb4-4520-\n0020: 39 66 32 38 2d 66 33 37 32 63 35 38 32 61 36 34 9f28-f372c582a64\n0030: 34 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 4.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:60533 req_id = 7b4655f2-223b-4991-9356-e1faccbc7ec0
30432025-04-30T06:43:02.037ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26838,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 99,\n 101,\n 102,\n 54,\n 102,\n 49,\n 102,\n 55,\n 45,\n 48,\n 98,\n 98,\n 52,\n 45,\n 52,\n 53,\n 50,\n 48,\n 45,\n 57,\n 102,\n 50,\n 56,\n 45,\n 102,\n 51,\n 55,\n 50,\n 99,\n 53,\n 56,\n 50,\n 97,\n 54,\n 52,\n 52,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60533 req_id = 7b4655f2-223b-4991-9356-e1faccbc7ec0
30442025-04-30T06:43:02.037ZDEBGtest_snapshot_unwind (store): query key key = cef6f1f7-0bb4-4520-9f28-f372c582a644.host
30452025-04-30T06:43:02.037ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60533 query = LowerQuery { name: LowerName(Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal.")), original: Query { name: Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("cef6f1f7-0bb4-4520-9f28-f372c582a644.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7b4655f2-223b-4991-9356-e1faccbc7ec0
30462025-04-30T06:43:02.096ZINFOtest_snapshot_unwind (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 63018 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:57125 req_id = 6011404e-6afa-4b05-897f-ee04bca0e06e response_code = 503 uri = /metrics/producers
30472025-04-30T06:43:02.097ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30482025-04-30T06:43:02.097ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:36829 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30492025-04-30T06:43:02.099ZDEBGtest_snapshot_unwind (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "6011404e-6afa-4b05-897f-ee04bca0e06e", "content-length": "133", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
30502025-04-30T06:43:02.105ZWARNtest_snapshot_unwind (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 598.371084ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"6011404e-6afa-4b05-897f-ee04bca0e06e\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:43:02 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"6011404e-6afa-4b05-897f-ee04bca0e06e\\" }" file = oximeter/producer/src/lib.rs:371
30512025-04-30T06:43:02.106ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30522025-04-30T06:43:02.107ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:36829 req_id = 9f1019e1-e413-41fd-84bf-51bcba197bdf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30532025-04-30T06:43:02.107ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 371 local_addr = [::1]:61186 method = GET remote_addr = [::1]:36829 req_id = 9f1019e1-e413-41fd-84bf-51bcba197bdf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30542025-04-30T06:43:02.879ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30552025-04-30T06:43:02.879ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:48828 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30562025-04-30T06:43:02.879ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:48445 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30572025-04-30T06:43:02.879ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:40699 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30582025-04-30T06:43:02.879ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:44446 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30592025-04-30T06:43:02.880ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:40917 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30602025-04-30T06:43:02.880ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:44656 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30612025-04-30T06:43:02.880ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:54444 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30622025-04-30T06:43:02.881ZDEBGtest_snapshot_unwind (producer-registration-task): will register with Nexus at 127.0.0.1:53046
30632025-04-30T06:43:02.951ZDEBGtest_snapshot_unwind (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:53046/metrics/producers
30642025-04-30T06:43:02.952ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30652025-04-30T06:43:02.953ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:52699
30662025-04-30T06:43:02.953ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:48828 req_id = 5565b98d-c56e-456e-b916-03e568577e4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30672025-04-30T06:43:02.953ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:48445 req_id = d5b42375-aebe-4c80-9fd3-0311c96def9c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30682025-04-30T06:43:02.953ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:40699 req_id = 10531b41-42f2-4b36-a08b-cba7b1707b33 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30692025-04-30T06:43:02.954ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:44446 req_id = 703def93-18b2-4ec7-994e-5a0a518c53d4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30702025-04-30T06:43:02.954ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:40917 req_id = 7a33d47a-8667-4626-ac9c-2fa097c73adb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30712025-04-30T06:43:02.954ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:44656 req_id = 44a8a28a-2dc3-4afc-b1f2-6622b345a682 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30722025-04-30T06:43:02.954ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:54444 req_id = 7c415a14-83d4-457a-890e-971a4f5a0ef3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30732025-04-30T06:43:02.954ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 1433 local_addr = [::1]:61186 method = GET remote_addr = [::1]:48828 req_id = 5565b98d-c56e-456e-b916-03e568577e4d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30742025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 1354 local_addr = [::1]:61186 method = GET remote_addr = [::1]:48445 req_id = d5b42375-aebe-4c80-9fd3-0311c96def9c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30752025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 1173 local_addr = [::1]:61186 method = GET remote_addr = [::1]:40699 req_id = 10531b41-42f2-4b36-a08b-cba7b1707b33 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30762025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 1020 local_addr = [::1]:61186 method = GET remote_addr = [::1]:44446 req_id = 703def93-18b2-4ec7-994e-5a0a518c53d4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30772025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 936 local_addr = [::1]:61186 method = GET remote_addr = [::1]:40917 req_id = 7a33d47a-8667-4626-ac9c-2fa097c73adb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30782025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 857 local_addr = [::1]:61186 method = GET remote_addr = [::1]:44656 req_id = 44a8a28a-2dc3-4afc-b1f2-6622b345a682 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30792025-04-30T06:43:02.955ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 777 local_addr = [::1]:61186 method = GET remote_addr = [::1]:54444 req_id = 7c415a14-83d4-457a-890e-971a4f5a0ef3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30802025-04-30T06:43:02.957ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52699 req_id = c57878c5-65df-4e70-9804-3507992b351f uri = /metrics/producers
30812025-04-30T06:43:02.959ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52699 req_id = c57878c5-65df-4e70-9804-3507992b351f resource = Database uri = /metrics/producers
30822025-04-30T06:43:02.959ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52699 req_id = c57878c5-65df-4e70-9804-3507992b351f roles = RoleSet { roles: {} } uri = /metrics/producers
30832025-04-30T06:43:02.960ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52699 req_id = c57878c5-65df-4e70-9804-3507992b351f resource = Database result = Ok(()) uri = /metrics/producers
30842025-04-30T06:43:02.961ZINFOtest_snapshot_unwind (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 version = 14
30852025-04-30T06:43:02.961ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30862025-04-30T06:43:02.969ZINFOtest_snapshot_unwind (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 11109 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52699 req_id = c57878c5-65df-4e70-9804-3507992b351f response_code = 503 uri = /metrics/producers
30872025-04-30T06:43:02.970ZDEBGtest_snapshot_unwind (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 208a51cf-96e0-4a2b-91aa-f434f7664e9b
30882025-04-30T06:43:02.971ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:44309 method = GET path = /info versions = all
30892025-04-30T06:43:02.972ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:44309 method = GET path = /producers versions = all
30902025-04-30T06:43:02.972ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:44309 method = DELETE path = /producers/{producer_id} versions = all
30912025-04-30T06:43:02.972ZDEBGtest_snapshot_unwind (dropshot): registered endpoint local_addr = [::1]:44309 method = GET path = /producers/{producer_id} versions = all
30922025-04-30T06:43:02.972ZINFOtest_snapshot_unwind (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:44309
30932025-04-30T06:43:02.972ZDEBGtest_snapshot_unwind (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:44309
30942025-04-30T06:43:02.973ZDEBGtest_snapshot_unwind (oximeter): registered USDT probes
30952025-04-30T06:43:02.973ZDEBGtest_snapshot_unwind (oximeter): contacting nexus
30962025-04-30T06:43:02.974ZDEBGtest_snapshot_unwind (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "c57878c5-65df-4e70-9804-3507992b351f", "content-length": "133", "date": "Wed, 30 Apr 2025 06:43:02 GMT"} })
30972025-04-30T06:43:02.975ZWARNtest_snapshot_unwind (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 1.46868603s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"c57878c5-65df-4e70-9804-3507992b351f\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:43:02 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"c57878c5-65df-4e70-9804-3507992b351f\\" }" file = oximeter/producer/src/lib.rs:371
30982025-04-30T06:43:03.506ZDEBGtest_snapshot_unwind (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:53046/metrics/collectors
30992025-04-30T06:43:03.520ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:52439
31002025-04-30T06:43:03.523ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52439 req_id = bddaa261-040f-43ba-a6ed-4a87c7d2563b uri = /metrics/collectors
31012025-04-30T06:43:03.523ZDEBGtest_snapshot_unwind (producer-registration-task): will register with Nexus at 127.0.0.1:53046
31022025-04-30T06:43:03.590ZDEBGtest_snapshot_unwind (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:53046/metrics/producers
31032025-04-30T06:43:03.593ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52439 req_id = bddaa261-040f-43ba-a6ed-4a87c7d2563b resource = Database uri = /metrics/collectors
31042025-04-30T06:43:03.593ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52439 req_id = bddaa261-040f-43ba-a6ed-4a87c7d2563b roles = RoleSet { roles: {} } uri = /metrics/collectors
31052025-04-30T06:43:03.594ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52439 req_id = bddaa261-040f-43ba-a6ed-4a87c7d2563b resource = Database result = Ok(()) uri = /metrics/collectors
31062025-04-30T06:43:03.595ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:58561
31072025-04-30T06:43:03.600ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:58561 req_id = 7beaae99-c706-461e-99c6-18c841f166a2 uri = /metrics/producers
31082025-04-30T06:43:03.601ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:58561 req_id = 7beaae99-c706-461e-99c6-18c841f166a2 resource = Database uri = /metrics/producers
31092025-04-30T06:43:03.601ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:58561 req_id = 7beaae99-c706-461e-99c6-18c841f166a2 roles = RoleSet { roles: {} } uri = /metrics/producers
31102025-04-30T06:43:03.602ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:58561 req_id = 7beaae99-c706-461e-99c6-18c841f166a2 resource = Database result = Ok(()) uri = /metrics/producers
31112025-04-30T06:43:03.605ZINFOtest_snapshot_unwind (nexus): registered new oximeter metric collection server address = [::1]:44309 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31122025-04-30T06:43:03.605ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 82136 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:52439 req_id = bddaa261-040f-43ba-a6ed-4a87c7d2563b response_code = 204 uri = /metrics/collectors
31132025-04-30T06:43:03.607ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors", status: 204, headers: {"x-request-id": "bddaa261-040f-43ba-a6ed-4a87c7d2563b", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
31142025-04-30T06:43:03.628ZINFOtest_snapshot_unwind (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31152025-04-30T06:43:03.628ZDEBGtest_snapshot_unwind (init_with_steps): Running step start_producer_server
31162025-04-30T06:43:03.628ZDEBGtest_snapshot_unwind: Starting test metric Producer Server
31172025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31182025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31192025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31202025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31212025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31222025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31232025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31242025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31252025-04-30T06:43:03.631ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:48860 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31262025-04-30T06:43:03.631ZINFOtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:48860 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31272025-04-30T06:43:03.632ZDEBGtest_snapshot_unwind (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:48860 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31282025-04-30T06:43:03.632ZINFOtest_snapshot_unwind (SledAgent): Created Simulated Crucible Server address = [::1]:48860 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31292025-04-30T06:43:03.632ZINFOtest_snapshot_unwind (nexus): inserting test physical disk model = test-model physical_disk_id = 162896b8-03f0-4bb0-aa03-fb464009fe73 serial = totally-unique-serial: 162896b8-03f0-4bb0-aa03-fb464009fe73 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31302025-04-30T06:43:03.632ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31312025-04-30T06:43:03.632ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31322025-04-30T06:43:03.633ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31332025-04-30T06:43:03.634ZINFOtest_snapshot_unwind (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31342025-04-30T06:43:03.679ZINFOtest_snapshot_unwind (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
31352025-04-30T06:43:03.688ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 87611 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:58561 req_id = 7beaae99-c706-461e-99c6-18c841f166a2 response_code = 201 uri = /metrics/producers
31362025-04-30T06:43:03.694ZINFOtest_snapshot_unwind (dropshot_internal): accepted connection local_addr = 127.0.0.1:53046 remote_addr = 127.0.0.1:64527
31372025-04-30T06:43:03.694ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31382025-04-30T06:43:03.694ZDEBGtest_snapshot_unwind (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "7beaae99-c706-461e-99c6-18c841f166a2", "content-length": "41", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
31392025-04-30T06:43:03.696ZDEBGtest_snapshot_unwind (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
31402025-04-30T06:43:03.696ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31412025-04-30T06:43:03.697ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31422025-04-30T06:43:03.697ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31432025-04-30T06:43:03.697ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31442025-04-30T06:43:03.698ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = ba532cfe-7078-4e83-999b-11110f3a4d6b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31452025-04-30T06:43:03.699ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:64527 req_id = 7990c7a3-aa91-48ff-836f-bd4d60a6c1e8 uri = /metrics/producers
31462025-04-30T06:43:03.704ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = ba532cfe-7078-4e83-999b-11110f3a4d6b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31472025-04-30T06:43:03.704ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = ba532cfe-7078-4e83-999b-11110f3a4d6b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31482025-04-30T06:43:03.705ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = ba532cfe-7078-4e83-999b-11110f3a4d6b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31492025-04-30T06:43:03.705ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:64527 req_id = 7990c7a3-aa91-48ff-836f-bd4d60a6c1e8 resource = Database uri = /metrics/producers
31502025-04-30T06:43:03.705ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:64527 req_id = 7990c7a3-aa91-48ff-836f-bd4d60a6c1e8 roles = RoleSet { roles: {} } uri = /metrics/producers
31512025-04-30T06:43:03.713ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:64527 req_id = 7990c7a3-aa91-48ff-836f-bd4d60a6c1e8 resource = Database result = Ok(()) uri = /metrics/producers
31522025-04-30T06:43:03.718ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31532025-04-30T06:43:03.718ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31542025-04-30T06:43:03.718ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31552025-04-30T06:43:03.718ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31562025-04-30T06:43:03.719ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31572025-04-30T06:43:03.733ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 35317 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = ba532cfe-7078-4e83-999b-11110f3a4d6b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31582025-04-30T06:43:03.735ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba532cfe-7078-4e83-999b-11110f3a4d6b", "content-length": "286", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
31592025-04-30T06:43:03.736ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31602025-04-30T06:43:03.736ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31612025-04-30T06:43:03.741ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31622025-04-30T06:43:03.747ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 4e35864d-9233-4d07-ae92-58612691bdaa uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31632025-04-30T06:43:03.747ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 4e35864d-9233-4d07-ae92-58612691bdaa resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31642025-04-30T06:43:03.747ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 4e35864d-9233-4d07-ae92-58612691bdaa roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31652025-04-30T06:43:03.747ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 4e35864d-9233-4d07-ae92-58612691bdaa resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31662025-04-30T06:43:03.759ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 14340 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 4e35864d-9233-4d07-ae92-58612691bdaa response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
31672025-04-30T06:43:03.760ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e35864d-9233-4d07-ae92-58612691bdaa", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
31682025-04-30T06:43:03.760ZDEBGtest_snapshot_unwind (oximeter-agent): registered new metric producer address = 127.0.0.1:49133 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
31692025-04-30T06:43:03.816ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
31702025-04-30T06:43:03.816ZINFOtest_snapshot_unwind (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31712025-04-30T06:43:03.816ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 117711 local_addr = 127.0.0.1:53046 method = POST remote_addr = 127.0.0.1:64527 req_id = 7990c7a3-aa91-48ff-836f-bd4d60a6c1e8 response_code = 201 uri = /metrics/producers
31722025-04-30T06:43:03.821ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 1 n_pruned_tasks = 0
31732025-04-30T06:43:03.832ZINFOtest_snapshot_unwind (nexus): upserting zpool physical_disk_id = 162896b8-03f0-4bb0-aa03-fb464009fe73 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = aed42bc8-b8a0-42a8-b67b-3a5a0927ed27
31742025-04-30T06:43:03.832ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31752025-04-30T06:43:03.832ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31762025-04-30T06:43:03.833ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31772025-04-30T06:43:03.836ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31782025-04-30T06:43:03.836ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31792025-04-30T06:43:03.836ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31802025-04-30T06:43:03.836ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31812025-04-30T06:43:03.836ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31822025-04-30T06:43:03.845ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31832025-04-30T06:43:03.847ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31842025-04-30T06:43:03.858ZINFOtest_snapshot_unwind (nexus): upserting Crucible dataset dataset_id = 463b6b79-e1f1-4c82-846e-8c24267f78be zpool_id = aed42bc8-b8a0-42a8-b67b-3a5a0927ed27
31852025-04-30T06:43:03.867ZINFOtest_snapshot_unwind (nexus): inserting test physical disk model = test-model physical_disk_id = 162896b8-03f0-4bb0-aa03-fb464009fe73 serial = totally-unique-serial: 162896b8-03f0-4bb0-aa03-fb464009fe73 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31862025-04-30T06:43:03.867ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31872025-04-30T06:43:03.867ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31882025-04-30T06:43:03.868ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31892025-04-30T06:43:03.877ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 162896b8-03f0-4bb0-aa03-fb464009fe73 (physical_disk), lookup_type: ById(162896b8-03f0-4bb0-aa03-fb464009fe73) }
31902025-04-30T06:43:03.877ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31912025-04-30T06:43:03.877ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31922025-04-30T06:43:03.877ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31932025-04-30T06:43:03.878ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31942025-04-30T06:43:03.887ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-04-30T06:43:03.892ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 162896b8-03f0-4bb0-aa03-fb464009fe73 (physical_disk), lookup_type: ById(162896b8-03f0-4bb0-aa03-fb464009fe73) } result = Ok(())
31962025-04-30T06:43:03.892ZINFOtest_snapshot_unwind (nexus): upserting zpool physical_disk_id = 162896b8-03f0-4bb0-aa03-fb464009fe73 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = aed42bc8-b8a0-42a8-b67b-3a5a0927ed27
31972025-04-30T06:43:03.892ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31982025-04-30T06:43:03.892ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
31992025-04-30T06:43:03.893ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32002025-04-30T06:43:03.901ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32012025-04-30T06:43:03.901ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32022025-04-30T06:43:03.901ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32032025-04-30T06:43:03.901ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32042025-04-30T06:43:03.902ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32052025-04-30T06:43:03.915ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32062025-04-30T06:43:03.917ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32072025-04-30T06:43:03.923ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32082025-04-30T06:43:03.924ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32092025-04-30T06:43:03.924ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32102025-04-30T06:43:03.924ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32112025-04-30T06:43:03.924ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32122025-04-30T06:43:03.933ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32132025-04-30T06:43:03.936ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32142025-04-30T06:43:03.936ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32152025-04-30T06:43:03.936ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32162025-04-30T06:43:03.936ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32172025-04-30T06:43:03.964ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32182025-04-30T06:43:03.964ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32192025-04-30T06:43:03.964ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32202025-04-30T06:43:03.964ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32212025-04-30T06:43:03.965ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32222025-04-30T06:43:03.965ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32232025-04-30T06:43:03.965ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32242025-04-30T06:43:03.965ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32252025-04-30T06:43:03.965ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32262025-04-30T06:43:03.965ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32272025-04-30T06:43:03.965ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32282025-04-30T06:43:03.974ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32292025-04-30T06:43:03.979ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32302025-04-30T06:43:03.979ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32312025-04-30T06:43:03.979ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32322025-04-30T06:43:03.982ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32332025-04-30T06:43:03.983ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32342025-04-30T06:43:03.987ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32352025-04-30T06:43:03.988ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32362025-04-30T06:43:03.988ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32372025-04-30T06:43:03.989ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32382025-04-30T06:43:03.990ZTRACtest_snapshot_unwind (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32392025-04-30T06:43:03.990ZTRACtest_snapshot_unwind (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32402025-04-30T06:43:03.997ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32412025-04-30T06:43:03.997ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
32422025-04-30T06:43:03.998ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32432025-04-30T06:43:03.999ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
32442025-04-30T06:43:03.999ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32452025-04-30T06:43:04.003ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c5 78 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .x..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57364 req_id = 5e8fdcce-1e96-4cbc-9ad0-287e396a804c
32462025-04-30T06:43:04.003ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50552,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57364 req_id = 5e8fdcce-1e96-4cbc-9ad0-287e396a804c
32472025-04-30T06:43:04.003ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
32482025-04-30T06:43:04.004ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32492025-04-30T06:43:04.004ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57364 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5e8fdcce-1e96-4cbc-9ad0-287e396a804c
32502025-04-30T06:43:04.005ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4248, tv_nsec: 804166362 } })
32512025-04-30T06:43:04.005ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
32522025-04-30T06:43:04.005ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: aa d4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60176 req_id = 40360d05-b9e1-4bbb-94fd-01ece090c62c
32532025-04-30T06:43:04.005ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43732,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60176 req_id = 40360d05-b9e1-4bbb-94fd-01ece090c62c
32542025-04-30T06:43:04.006ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32552025-04-30T06:43:04.006ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60176 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 40360d05-b9e1-4bbb-94fd-01ece090c62c
32562025-04-30T06:43:04.060ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32572025-04-30T06:43:04.060ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 22 1c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c "............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65327 req_id = 641ddc97-d7af-4f09-bab8-6b746eb716df
32582025-04-30T06:43:04.061ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8732,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65327 req_id = 641ddc97-d7af-4f09-bab8-6b746eb716df
32592025-04-30T06:43:04.061ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
32602025-04-30T06:43:04.061ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65327 req_id = 641ddc97-d7af-4f09-bab8-6b746eb716df
32612025-04-30T06:43:04.061ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
32622025-04-30T06:43:04.061ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
32632025-04-30T06:43:04.061ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
32642025-04-30T06:43:04.063ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d4b561a-d529-481a-89af-fb7b7cd87d58", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32652025-04-30T06:43:04.063ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
32662025-04-30T06:43:04.065ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da42469-2457-48c6-b972-2060cf1cf2e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32672025-04-30T06:43:04.065ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
32682025-04-30T06:43:04.068ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ded9f438-d84e-4afc-9b6a-ea65d6b01ad6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32692025-04-30T06:43:04.068ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
32702025-04-30T06:43:04.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3c09e9-b92a-48e4-a849-2c7c12ccf18f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32712025-04-30T06:43:04.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
32722025-04-30T06:43:04.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "238dc603-5f03-4a69-adb4-5676daca24f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32732025-04-30T06:43:04.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
32742025-04-30T06:43:04.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1858215c-e8d9-493e-947b-e437ecfb887e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32752025-04-30T06:43:04.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
32762025-04-30T06:43:04.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "319b7363-e72c-434f-9457-3b0311aa95ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32772025-04-30T06:43:04.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
32782025-04-30T06:43:04.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b94ac5-5a57-472d-85da-6df32790ba03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32792025-04-30T06:43:04.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
32802025-04-30T06:43:04.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c063cc0-c230-4c1c-b886-033a0c4aedd6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32812025-04-30T06:43:04.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
32822025-04-30T06:43:04.097ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a5703f-d9a6-4227-92cf-b79a20edc304", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32832025-04-30T06:43:04.098ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
32842025-04-30T06:43:04.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f554ed9-82e0-4b66-974b-6463fd8aa5a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32852025-04-30T06:43:04.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
32862025-04-30T06:43:04.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1538d88e-3082-47c7-96e6-762fd2b85fd1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32872025-04-30T06:43:04.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
32882025-04-30T06:43:04.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e563804e-74a3-4dcb-b6fd-2405bfe30dbb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32892025-04-30T06:43:04.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
32902025-04-30T06:43:04.116ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8cb9421-05c8-4ce7-a9a6-c7659c5d07cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32912025-04-30T06:43:04.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
32922025-04-30T06:43:04.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94759a9e-1b6a-44e1-8675-3c12ab63ce27", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32932025-04-30T06:43:04.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
32942025-04-30T06:43:04.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "131570d7-a8b3-4619-b746-793287fa38f5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32952025-04-30T06:43:04.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
32962025-04-30T06:43:04.129ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
32972025-04-30T06:43:04.133ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ac5a0c-0a6f-4968-abb6-3d21ccba7b88", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
32982025-04-30T06:43:04.133ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
32992025-04-30T06:43:04.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b0073fe-6f15-427d-968d-8a26b1881d0b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33002025-04-30T06:43:04.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
33012025-04-30T06:43:04.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80522246-61eb-4d3f-ab93-8b4aa8ed6151", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33022025-04-30T06:43:04.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
33032025-04-30T06:43:04.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e10f4335-c480-441b-97c6-838af86c4738", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33042025-04-30T06:43:04.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
33052025-04-30T06:43:04.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aab2a5a4-2a60-4c8b-8e67-3687bae371c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33062025-04-30T06:43:04.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
33072025-04-30T06:43:04.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20efd1d2-7c76-43ba-bcdc-d1641c13c53d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33082025-04-30T06:43:04.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
33092025-04-30T06:43:04.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd5bd729-8d53-45de-9298-f9a2dac5ff56", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33102025-04-30T06:43:04.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
33112025-04-30T06:43:04.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae6d274-e447-4f23-9cdb-79085bdf6a5d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33122025-04-30T06:43:04.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
33132025-04-30T06:43:04.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9e3d432-5efa-4b16-93fd-763653c139a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33142025-04-30T06:43:04.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
33152025-04-30T06:43:04.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7f7d9d0-add8-4217-983d-ae751f6103dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33162025-04-30T06:43:04.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
33172025-04-30T06:43:04.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba2343c8-1eab-4eb0-9fab-88459d0da4e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33182025-04-30T06:43:04.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
33192025-04-30T06:43:04.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eab2d404-fa90-40ef-892b-fc20af34f21a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33202025-04-30T06:43:04.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
33212025-04-30T06:43:04.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b301628-3e92-4ed2-8975-9b74fac06cce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33222025-04-30T06:43:04.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
33232025-04-30T06:43:04.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5982d70e-94fb-48d8-a901-816f96a22adc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33242025-04-30T06:43:04.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
33252025-04-30T06:43:04.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e162cc-bac0-4535-9ae1-ef70da2f4ec4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33262025-04-30T06:43:04.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
33272025-04-30T06:43:04.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80022a5e-289d-427a-af85-cd187d1c3c9f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33282025-04-30T06:43:04.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
33292025-04-30T06:43:04.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09110b0b-2104-4a02-bf16-80f9bc9334f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33302025-04-30T06:43:04.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
33312025-04-30T06:43:04.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6162203f-878e-46cf-8d0c-e50f6a3f5e3b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33322025-04-30T06:43:04.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
33332025-04-30T06:43:04.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e172e5f2-59f1-49d2-8f29-c83cae1f3058", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33342025-04-30T06:43:04.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
33352025-04-30T06:43:04.180ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33362025-04-30T06:43:04.180ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33372025-04-30T06:43:04.180ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33382025-04-30T06:43:04.180ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
33392025-04-30T06:43:04.181ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33402025-04-30T06:43:04.184ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33412025-04-30T06:43:04.193ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33422025-04-30T06:43:04.193ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33432025-04-30T06:43:04.193ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
33442025-04-30T06:43:04.194ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33452025-04-30T06:43:04.199ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33462025-04-30T06:43:04.199ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
33472025-04-30T06:43:04.200ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33482025-04-30T06:43:04.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "882f818c-45a8-4b08-983e-89446909c1b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33492025-04-30T06:43:04.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
33502025-04-30T06:43:04.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e31110-be85-4ed5-8945-d8a30f5ef64e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33512025-04-30T06:43:04.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
33522025-04-30T06:43:04.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75472728-6dd8-4b14-80e4-4ba214469e7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33532025-04-30T06:43:04.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
33542025-04-30T06:43:04.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a661ed84-7da3-4ac8-a5e0-a5ea6211d40b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33552025-04-30T06:43:04.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
33562025-04-30T06:43:04.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1c741f8-bb05-4f96-82df-81012213203e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33572025-04-30T06:43:04.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
33582025-04-30T06:43:04.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5530aec-06ed-418d-8f1a-d3833b36817e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33592025-04-30T06:43:04.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
33602025-04-30T06:43:04.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce7083d4-490a-4016-8456-3a45f2fa325f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33612025-04-30T06:43:04.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
33622025-04-30T06:43:04.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96aa417-f7ef-4b47-bd8d-766ab80456d9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33632025-04-30T06:43:04.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
33642025-04-30T06:43:04.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f71f92-c260-4622-8665-e860593f2da3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33652025-04-30T06:43:04.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
33662025-04-30T06:43:04.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "977539af-e767-4eab-9b67-6637711c5bbf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33672025-04-30T06:43:04.227ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33682025-04-30T06:43:04.227ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33692025-04-30T06:43:04.227ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33702025-04-30T06:43:04.227ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33712025-04-30T06:43:04.227ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33722025-04-30T06:43:04.231ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33732025-04-30T06:43:04.237ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33742025-04-30T06:43:04.237ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33752025-04-30T06:43:04.237ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33762025-04-30T06:43:04.237ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33772025-04-30T06:43:04.237ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33782025-04-30T06:43:04.238ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33792025-04-30T06:43:04.242ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33802025-04-30T06:43:04.246ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33812025-04-30T06:43:04.247ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33822025-04-30T06:43:04.247ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33832025-04-30T06:43:04.247ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33842025-04-30T06:43:04.252ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
33852025-04-30T06:43:04.311ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
33862025-04-30T06:43:04.312ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
33872025-04-30T06:43:04.313ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33882025-04-30T06:43:04.313ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33892025-04-30T06:43:04.313ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33902025-04-30T06:43:04.313ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
33912025-04-30T06:43:04.314ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33922025-04-30T06:43:04.315ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:47684 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33932025-04-30T06:43:04.316ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:47684 req_id = 17219fdf-4b9b-4fc9-865a-95b074f4e514 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33942025-04-30T06:43:04.316ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 851 local_addr = [::1]:45714 method = GET remote_addr = [::1]:47684 req_id = 17219fdf-4b9b-4fc9-865a-95b074f4e514 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33952025-04-30T06:43:04.317ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "17219fdf-4b9b-4fc9-865a-95b074f4e514", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:03 GMT"} })
33962025-04-30T06:43:04.319ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33972025-04-30T06:43:04.319ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33982025-04-30T06:43:04.319ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
33992025-04-30T06:43:04.324ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34002025-04-30T06:43:04.325ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34012025-04-30T06:43:04.325ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34022025-04-30T06:43:04.325ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34032025-04-30T06:43:04.326ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34042025-04-30T06:43:04.326ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34052025-04-30T06:43:04.334ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34062025-04-30T06:43:04.334ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34072025-04-30T06:43:04.334ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
34082025-04-30T06:43:04.335ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34092025-04-30T06:43:04.342ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34102025-04-30T06:43:04.347ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34112025-04-30T06:43:04.347ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34122025-04-30T06:43:04.347ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34132025-04-30T06:43:04.348ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34142025-04-30T06:43:04.350ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34152025-04-30T06:43:04.350ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
34162025-04-30T06:43:04.351ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34172025-04-30T06:43:04.429ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
34182025-04-30T06:43:04.481ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34192025-04-30T06:43:04.481ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34202025-04-30T06:43:04.481ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34212025-04-30T06:43:04.481ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
34222025-04-30T06:43:04.482ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34232025-04-30T06:43:04.485ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34242025-04-30T06:43:04.490ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34252025-04-30T06:43:04.490ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34262025-04-30T06:43:04.491ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
34272025-04-30T06:43:04.491ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34282025-04-30T06:43:04.529ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34292025-04-30T06:43:04.529ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
34302025-04-30T06:43:04.530ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34312025-04-30T06:43:04.530ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34322025-04-30T06:43:04.531ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34332025-04-30T06:43:04.531ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 565.795591ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34342025-04-30T06:43:04.531ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34352025-04-30T06:43:04.531ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34362025-04-30T06:43:04.531ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34372025-04-30T06:43:04.531ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34382025-04-30T06:43:04.531ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34392025-04-30T06:43:04.532ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34402025-04-30T06:43:04.535ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34412025-04-30T06:43:04.539ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34422025-04-30T06:43:04.539ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34432025-04-30T06:43:04.539ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34442025-04-30T06:43:04.540ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34452025-04-30T06:43:04.543ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
34462025-04-30T06:43:04.543ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34472025-04-30T06:43:04.544ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 08 54 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .T..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49267 req_id = 56bfa402-9cf5-40bb-bbbb-a103bf556e9e
34482025-04-30T06:43:04.544ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2132,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49267 req_id = 56bfa402-9cf5-40bb-bbbb-a103bf556e9e
34492025-04-30T06:43:04.545ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
34502025-04-30T06:43:04.545ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34512025-04-30T06:43:04.545ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49267 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 56bfa402-9cf5-40bb-bbbb-a103bf556e9e
34522025-04-30T06:43:04.545ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4249, tv_nsec: 344578208 } })
34532025-04-30T06:43:04.545ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
34542025-04-30T06:43:04.546ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 81 0d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52525 req_id = 2d9cce87-38a2-443b-90b8-d8e005713665
34552025-04-30T06:43:04.546ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52525 req_id = 2d9cce87-38a2-443b-90b8-d8e005713665
34562025-04-30T06:43:04.546ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34572025-04-30T06:43:04.546ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52525 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2d9cce87-38a2-443b-90b8-d8e005713665
34582025-04-30T06:43:04.583ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34592025-04-30T06:43:04.584ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba ec 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49774 req_id = 2926094d-14f5-453b-84da-4b6554816e9e
34602025-04-30T06:43:04.584ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47852,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49774 req_id = 2926094d-14f5-453b-84da-4b6554816e9e
34612025-04-30T06:43:04.584ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
34622025-04-30T06:43:04.584ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47852, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49774 req_id = 2926094d-14f5-453b-84da-4b6554816e9e
34632025-04-30T06:43:04.584ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
34642025-04-30T06:43:04.584ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
34652025-04-30T06:43:04.584ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
34662025-04-30T06:43:04.586ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "562863c6-6047-455c-bc96-4625a0fa165a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34672025-04-30T06:43:04.586ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
34682025-04-30T06:43:04.588ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "694e66da-5dd1-4f41-bfb7-ff3b3f7ba45f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34692025-04-30T06:43:04.588ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
34702025-04-30T06:43:04.590ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8182a004-427f-4fca-9643-39725dea83b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34712025-04-30T06:43:04.590ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
34722025-04-30T06:43:04.592ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8787d7d-e37f-47ad-ab68-7668214c37d5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34732025-04-30T06:43:04.592ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
34742025-04-30T06:43:04.594ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ce40d47-e60f-40ac-a3a7-8dfd4f465172", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34752025-04-30T06:43:04.594ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
34762025-04-30T06:43:04.596ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ee4c864-f46f-461d-8e79-f3d82d3ea678", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34772025-04-30T06:43:04.596ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
34782025-04-30T06:43:04.598ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac4b80f-ee38-4a36-ba6f-d953b1010138", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34792025-04-30T06:43:04.598ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
34802025-04-30T06:43:04.606ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03512782-6519-4509-9080-70171a71855a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34812025-04-30T06:43:04.606ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
34822025-04-30T06:43:04.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f72958cc-ddf2-4e6b-85cc-b3ed78e45458", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34832025-04-30T06:43:04.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
34842025-04-30T06:43:04.611ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ddfb1b-fb7e-4656-a980-a43eb311b58c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34852025-04-30T06:43:04.611ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
34862025-04-30T06:43:04.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69d866b-870e-497f-96cc-8d1845c1797c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34872025-04-30T06:43:04.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
34882025-04-30T06:43:04.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b47ce99d-0b7b-4004-9f20-85c96a314985", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34892025-04-30T06:43:04.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
34902025-04-30T06:43:04.615ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
34912025-04-30T06:43:04.617ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a26a5b8-e77f-4871-a893-1fec83462a31", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34922025-04-30T06:43:04.617ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
34932025-04-30T06:43:04.620ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88f7dffc-f64b-4984-8c2d-a360b195170b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34942025-04-30T06:43:04.620ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
34952025-04-30T06:43:04.623ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27186be1-32fe-4212-b00c-b5c36ba69a5c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34962025-04-30T06:43:04.623ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
34972025-04-30T06:43:04.625ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1351cba6-0ed1-42f2-b94e-6693e0aa82a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
34982025-04-30T06:43:04.625ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
34992025-04-30T06:43:04.627ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc17f96d-5427-438b-bf21-e4308611ca94", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35002025-04-30T06:43:04.627ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
35012025-04-30T06:43:04.629ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c4e925a-28d1-4abc-9340-ec586138c34e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35022025-04-30T06:43:04.629ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
35032025-04-30T06:43:04.631ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f4fc63a-6922-49aa-8fea-4a0457226704", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35042025-04-30T06:43:04.631ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
35052025-04-30T06:43:04.633ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7184d7cc-82d4-4f27-ad5c-bb7b1acfab63", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35062025-04-30T06:43:04.633ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
35072025-04-30T06:43:04.634ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfd13e06-ccd9-4196-be52-1899399ef89a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35082025-04-30T06:43:04.634ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
35092025-04-30T06:43:04.636ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba28973e-5b0c-4f1e-80a0-764ddc965110", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35102025-04-30T06:43:04.636ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
35112025-04-30T06:43:04.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9341c72-62b8-48b7-b3ee-87888c24fe27", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35122025-04-30T06:43:04.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
35132025-04-30T06:43:04.640ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "690ebc26-34de-4909-81b3-f14fb422e235", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35142025-04-30T06:43:04.640ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
35152025-04-30T06:43:04.643ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87173947-0359-4ef8-9894-0ce958b7b191", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35162025-04-30T06:43:04.643ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
35172025-04-30T06:43:04.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1b7e06-a789-461b-bba5-8b31266d58b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35182025-04-30T06:43:04.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
35192025-04-30T06:43:04.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef16b2a3-5a5e-4215-82a8-c91bd21e9b01", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35202025-04-30T06:43:04.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
35212025-04-30T06:43:04.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03fa200a-abe3-470e-817f-b00f511a572b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35222025-04-30T06:43:04.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
35232025-04-30T06:43:04.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd5674a-7dac-4031-8844-5e8d9cc0a00d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35242025-04-30T06:43:04.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
35252025-04-30T06:43:04.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b2b0211-8cc4-4551-94a4-1ca1fe823b58", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35262025-04-30T06:43:04.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
35272025-04-30T06:43:04.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baa61530-7726-4678-a4ee-8d7cc9f166cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35282025-04-30T06:43:04.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
35292025-04-30T06:43:04.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b10d11-3bd4-4d04-9656-ed443aaad3bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35302025-04-30T06:43:04.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
35312025-04-30T06:43:04.667ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35322025-04-30T06:43:04.667ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35332025-04-30T06:43:04.667ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35342025-04-30T06:43:04.667ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
35352025-04-30T06:43:04.669ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35362025-04-30T06:43:04.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42df581e-e7b2-4f94-aca4-d342675a4884", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35372025-04-30T06:43:04.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
35382025-04-30T06:43:04.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee03acae-7163-40c4-9034-e2e30ad0ae4d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35392025-04-30T06:43:04.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
35402025-04-30T06:43:04.673ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35412025-04-30T06:43:04.682ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35422025-04-30T06:43:04.682ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35432025-04-30T06:43:04.682ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
35442025-04-30T06:43:04.682ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35452025-04-30T06:43:04.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "148e2024-4300-4910-80b6-f1b377831d78", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35462025-04-30T06:43:04.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
35472025-04-30T06:43:04.687ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35482025-04-30T06:43:04.687ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
35492025-04-30T06:43:04.688ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35502025-04-30T06:43:04.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "554bc283-823e-456e-a4d5-b2cbb8317f79", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35512025-04-30T06:43:04.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
35522025-04-30T06:43:04.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2349b7f-cfb0-4518-acff-a84010e3aabd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35532025-04-30T06:43:04.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
35542025-04-30T06:43:04.706ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47f53b0f-19c5-49a2-b541-f9c3e2d08dd8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35552025-04-30T06:43:04.706ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
35562025-04-30T06:43:04.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b677c47d-56f9-4bfb-8622-54a87fae2381", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35572025-04-30T06:43:04.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
35582025-04-30T06:43:04.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51ded0ab-00a8-4557-b31f-0db97e492a84", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35592025-04-30T06:43:04.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
35602025-04-30T06:43:04.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd1c1a1-9448-4228-842f-a6367754b149", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35612025-04-30T06:43:04.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
35622025-04-30T06:43:04.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f8e84e-02d8-461c-9821-b0b1f08da2f6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35632025-04-30T06:43:04.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
35642025-04-30T06:43:04.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef482057-7dfe-4875-b35b-5dd112f7d70b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35652025-04-30T06:43:04.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
35662025-04-30T06:43:04.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccfa0b47-e3ab-4ac4-8b5d-2cfb2cbfa5c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35672025-04-30T06:43:04.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
35682025-04-30T06:43:04.727ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26c6e28f-e398-4ff7-b206-7abd1d4411ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35692025-04-30T06:43:04.727ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35702025-04-30T06:43:04.727ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35712025-04-30T06:43:04.728ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35722025-04-30T06:43:04.728ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35732025-04-30T06:43:04.728ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35742025-04-30T06:43:04.731ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35752025-04-30T06:43:04.738ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35762025-04-30T06:43:04.738ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35772025-04-30T06:43:04.738ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35782025-04-30T06:43:04.738ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35792025-04-30T06:43:04.738ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35802025-04-30T06:43:04.739ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35812025-04-30T06:43:04.742ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35822025-04-30T06:43:04.747ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35832025-04-30T06:43:04.748ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35842025-04-30T06:43:04.748ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35852025-04-30T06:43:04.749ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35862025-04-30T06:43:04.820ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
35872025-04-30T06:43:04.820ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
35882025-04-30T06:43:04.821ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
35892025-04-30T06:43:04.822ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:47131 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35902025-04-30T06:43:04.823ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:47131 req_id = 4209a6c5-cf3e-4c7d-8882-2edb6bee3cf3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35912025-04-30T06:43:04.824ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 974 local_addr = [::1]:45714 method = GET remote_addr = [::1]:47131 req_id = 4209a6c5-cf3e-4c7d-8882-2edb6bee3cf3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35922025-04-30T06:43:04.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4209a6c5-cf3e-4c7d-8882-2edb6bee3cf3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
35932025-04-30T06:43:04.827ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35942025-04-30T06:43:04.827ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35952025-04-30T06:43:04.827ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
35962025-04-30T06:43:04.829ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
35972025-04-30T06:43:04.829ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35982025-04-30T06:43:04.829ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35992025-04-30T06:43:04.829ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36002025-04-30T06:43:04.830ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36012025-04-30T06:43:04.833ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36022025-04-30T06:43:04.838ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36032025-04-30T06:43:04.838ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36042025-04-30T06:43:04.838ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36052025-04-30T06:43:04.839ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36062025-04-30T06:43:04.876ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36072025-04-30T06:43:04.876ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36082025-04-30T06:43:04.876ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36092025-04-30T06:43:04.876ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
36102025-04-30T06:43:04.877ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36112025-04-30T06:43:04.885ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36122025-04-30T06:43:04.885ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36132025-04-30T06:43:04.885ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 354.653253ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36142025-04-30T06:43:04.886ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36152025-04-30T06:43:04.886ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36162025-04-30T06:43:04.886ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36172025-04-30T06:43:04.886ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36182025-04-30T06:43:04.886ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36192025-04-30T06:43:04.887ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36202025-04-30T06:43:04.889ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36212025-04-30T06:43:04.902ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36222025-04-30T06:43:04.902ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36232025-04-30T06:43:04.902ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
36242025-04-30T06:43:04.904ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36252025-04-30T06:43:04.908ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36262025-04-30T06:43:04.925ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36272025-04-30T06:43:04.925ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36282025-04-30T06:43:04.925ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36292025-04-30T06:43:04.925ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36302025-04-30T06:43:04.928ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36312025-04-30T06:43:04.928ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
36322025-04-30T06:43:04.929ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36332025-04-30T06:43:04.931ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
36342025-04-30T06:43:04.932ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36352025-04-30T06:43:04.932ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 be 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56236 req_id = 5058a705-d562-472f-8fa5-0d99f9f881b8
36362025-04-30T06:43:04.932ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 958,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56236 req_id = 5058a705-d562-472f-8fa5-0d99f9f881b8
36372025-04-30T06:43:04.932ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
36382025-04-30T06:43:04.933ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36392025-04-30T06:43:04.933ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56236 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5058a705-d562-472f-8fa5-0d99f9f881b8
36402025-04-30T06:43:04.933ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4249, tv_nsec: 732686557 } })
36412025-04-30T06:43:04.933ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
36422025-04-30T06:43:04.934ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 30 b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 0...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64879 req_id = d8fa9f5c-8495-464b-8095-d93ef55b3a47
36432025-04-30T06:43:04.934ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64879 req_id = d8fa9f5c-8495-464b-8095-d93ef55b3a47
36442025-04-30T06:43:04.934ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36452025-04-30T06:43:04.934ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64879 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d8fa9f5c-8495-464b-8095-d93ef55b3a47
36462025-04-30T06:43:05.021ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36472025-04-30T06:43:05.044ZTRACtest_snapshot_unwind (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
36482025-04-30T06:43:05.045ZTRACtest_snapshot_unwind (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
36492025-04-30T06:43:05.045ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dc d3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57133 req_id = 2e829a2d-756e-49bb-ab3c-50008c15529a
36502025-04-30T06:43:05.045ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56531,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57133 req_id = 2e829a2d-756e-49bb-ab3c-50008c15529a
36512025-04-30T06:43:05.045ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
36522025-04-30T06:43:05.045ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56531, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57133 req_id = 2e829a2d-756e-49bb-ab3c-50008c15529a
36532025-04-30T06:43:05.046ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
36542025-04-30T06:43:05.046ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
36552025-04-30T06:43:05.046ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
36562025-04-30T06:43:05.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "adacaf83-0367-47eb-9c4f-81f5c0e4de1a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36572025-04-30T06:43:05.048ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
36582025-04-30T06:43:05.050ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f88016-df00-42db-9ecb-ff471cbf92bc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36592025-04-30T06:43:05.051ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
36602025-04-30T06:43:05.053ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c12ff33e-54db-4f71-8c64-e32b6b36074d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36612025-04-30T06:43:05.053ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
36622025-04-30T06:43:05.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "395bbc8c-9f61-4186-b325-26c036c5974e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36632025-04-30T06:43:05.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
36642025-04-30T06:43:05.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa8c9380-1406-4b8f-bae1-bc69070ab998", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36652025-04-30T06:43:05.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
36662025-04-30T06:43:05.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c9727b-42b8-491d-8c1f-b90ee49f3f19", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36672025-04-30T06:43:05.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
36682025-04-30T06:43:05.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed9d154-e449-463e-aec4-7671d36c84d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36692025-04-30T06:43:05.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
36702025-04-30T06:43:05.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e2590a-76bf-4aae-85ba-83419736d616", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36712025-04-30T06:43:05.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
36722025-04-30T06:43:05.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cada7628-f12c-4815-a5c6-8c6424b6d99e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36732025-04-30T06:43:05.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
36742025-04-30T06:43:05.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2114c81-6478-4cdd-8f39-a7f4713463ff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36752025-04-30T06:43:05.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
36762025-04-30T06:43:05.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3543fec-221a-487a-8f1e-376e13ab90c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36772025-04-30T06:43:05.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
36782025-04-30T06:43:05.091ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b4fcbf-ab56-4aaa-beed-edf6e4c4c313", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36792025-04-30T06:43:05.092ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
36802025-04-30T06:43:05.096ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "578fff12-df2d-445c-9098-3b2043964c3f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36812025-04-30T06:43:05.096ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
36822025-04-30T06:43:05.102ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
36832025-04-30T06:43:05.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1cd614a-446d-4c69-855b-3a3f4d32cf27", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36842025-04-30T06:43:05.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
36852025-04-30T06:43:05.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32405a4a-17aa-4f97-a5fe-22900c31b751", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36862025-04-30T06:43:05.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
36872025-04-30T06:43:05.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7044c151-da3f-4e94-b1d5-217971d64b77", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36882025-04-30T06:43:05.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
36892025-04-30T06:43:05.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fc75733-d13b-44ca-9cc8-bbb19d1b235a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36902025-04-30T06:43:05.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
36912025-04-30T06:43:05.116ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5b3719-d945-4786-9f82-5f1ccd97ce02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36922025-04-30T06:43:05.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
36932025-04-30T06:43:05.120ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ce1e6f3-1ece-4499-a2db-ad504142d9ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36942025-04-30T06:43:05.120ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
36952025-04-30T06:43:05.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e37cba-2a5f-46de-a3f6-5df5653e36b8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36962025-04-30T06:43:05.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
36972025-04-30T06:43:05.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aab2fe9b-0bca-47f2-ac22-2efcb2af2e7d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
36982025-04-30T06:43:05.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
36992025-04-30T06:43:05.125ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea9e1d75-ebf5-4a61-bd29-080e9da4c8b7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37002025-04-30T06:43:05.125ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
37012025-04-30T06:43:05.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dba693d3-8929-4721-b4a9-712ebb7338a5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37022025-04-30T06:43:05.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
37032025-04-30T06:43:05.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6759b068-1e40-4db6-aa0c-7fac7376fb50", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37042025-04-30T06:43:05.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
37052025-04-30T06:43:05.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af94d48f-02f5-4dc6-8491-188e649edfa5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37062025-04-30T06:43:05.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
37072025-04-30T06:43:05.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f6a7cc1-b777-4868-b5cb-e97936699430", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37082025-04-30T06:43:05.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
37092025-04-30T06:43:05.153ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37102025-04-30T06:43:05.153ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37112025-04-30T06:43:05.153ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37122025-04-30T06:43:05.153ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
37132025-04-30T06:43:05.154ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37142025-04-30T06:43:05.156ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37152025-04-30T06:43:05.161ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37162025-04-30T06:43:05.162ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37172025-04-30T06:43:05.162ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
37182025-04-30T06:43:05.162ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37192025-04-30T06:43:05.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be7a5bbd-978d-4eeb-afcd-ccb2883fd394", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37202025-04-30T06:43:05.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
37212025-04-30T06:43:05.165ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37222025-04-30T06:43:05.165ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
37232025-04-30T06:43:05.165ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37242025-04-30T06:43:05.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55f3a709-7336-4dc8-b7e5-d6038ae0df37", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37252025-04-30T06:43:05.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
37262025-04-30T06:43:05.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0df232c-9ce4-46c4-ab99-68c933940bf4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37272025-04-30T06:43:05.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
37282025-04-30T06:43:05.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d218b10a-0aa7-4026-bf83-6d3cacc59323", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37292025-04-30T06:43:05.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
37302025-04-30T06:43:05.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffc08adb-0667-499f-951a-70e5eed0339a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37312025-04-30T06:43:05.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
37322025-04-30T06:43:05.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd2c5c09-2ec4-4fc7-84df-ca88fb664fd7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37332025-04-30T06:43:05.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
37342025-04-30T06:43:05.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ed05ec1-c7af-4141-85e1-2c8be5e8eb4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37352025-04-30T06:43:05.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
37362025-04-30T06:43:05.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "051c4616-73d3-4481-89f7-953271237c57", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37372025-04-30T06:43:05.190ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
37382025-04-30T06:43:05.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f42ff84-f7e8-4c31-ba89-a92941bd7657", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37392025-04-30T06:43:05.192ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
37402025-04-30T06:43:05.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "639cf579-da58-490c-9e20-b2e59181ea1f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37412025-04-30T06:43:05.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
37422025-04-30T06:43:05.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cfd3cd1-2b62-454e-9cab-e277a1e7a42e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37432025-04-30T06:43:05.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
37442025-04-30T06:43:05.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f14f06b7-37d3-47c4-a876-bc084c1d1085", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37452025-04-30T06:43:05.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
37462025-04-30T06:43:05.212ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
37472025-04-30T06:43:05.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74a3a744-6c2c-488c-8cbf-32731bd2d898", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37482025-04-30T06:43:05.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
37492025-04-30T06:43:05.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2746bd8-02b7-4f69-97c7-acb7b8934dcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37502025-04-30T06:43:05.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
37512025-04-30T06:43:05.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b15ea746-8a15-4ace-9593-c55cf954adac", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37522025-04-30T06:43:05.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
37532025-04-30T06:43:05.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d056591-dc83-49b0-824f-30a3a446a0c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37542025-04-30T06:43:05.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
37552025-04-30T06:43:05.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9283a4e-ffed-48a2-a439-a1f0dcbc8c2b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37562025-04-30T06:43:05.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
37572025-04-30T06:43:05.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c787008-37a6-453a-8c97-eb479ca8b77f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37582025-04-30T06:43:05.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
37592025-04-30T06:43:05.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ecc665a-302e-4da4-98a1-6b0e1a59f7b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37602025-04-30T06:43:05.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37612025-04-30T06:43:05.231ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37622025-04-30T06:43:05.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37632025-04-30T06:43:05.231ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37642025-04-30T06:43:05.232ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37652025-04-30T06:43:05.244ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37662025-04-30T06:43:05.248ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37672025-04-30T06:43:05.248ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37682025-04-30T06:43:05.248ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37692025-04-30T06:43:05.248ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37702025-04-30T06:43:05.248ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37712025-04-30T06:43:05.248ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37722025-04-30T06:43:05.270ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37732025-04-30T06:43:05.276ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37742025-04-30T06:43:05.276ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37752025-04-30T06:43:05.276ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37762025-04-30T06:43:05.276ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37772025-04-30T06:43:05.286ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37782025-04-30T06:43:05.286ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37792025-04-30T06:43:05.286ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37802025-04-30T06:43:05.286ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
37812025-04-30T06:43:05.286ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37822025-04-30T06:43:05.352ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
37832025-04-30T06:43:05.352ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
37842025-04-30T06:43:05.353ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:47526 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
37852025-04-30T06:43:05.354ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:47526 req_id = f9f94afe-44d2-447c-bd7c-6d04d0b3b6d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
37862025-04-30T06:43:05.357ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 3235 local_addr = [::1]:45714 method = GET remote_addr = [::1]:47526 req_id = f9f94afe-44d2-447c-bd7c-6d04d0b3b6d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
37872025-04-30T06:43:05.359ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37882025-04-30T06:43:05.366ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37892025-04-30T06:43:05.366ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37902025-04-30T06:43:05.366ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
37912025-04-30T06:43:05.367ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37922025-04-30T06:43:05.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f94afe-44d2-447c-bd7c-6d04d0b3b6d9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:04 GMT"} })
37932025-04-30T06:43:05.372ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
37942025-04-30T06:43:05.372ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
37952025-04-30T06:43:05.372ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
37962025-04-30T06:43:05.374ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
37972025-04-30T06:43:05.374ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37982025-04-30T06:43:05.374ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37992025-04-30T06:43:05.374ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38002025-04-30T06:43:05.374ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38012025-04-30T06:43:05.386ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38022025-04-30T06:43:05.386ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
38032025-04-30T06:43:05.387ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38042025-04-30T06:43:05.388ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38052025-04-30T06:43:05.405ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38062025-04-30T06:43:05.405ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38072025-04-30T06:43:05.405ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38082025-04-30T06:43:05.406ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38092025-04-30T06:43:05.472ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38102025-04-30T06:43:05.473ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38112025-04-30T06:43:05.473ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 587.923149ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38122025-04-30T06:43:05.474ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38132025-04-30T06:43:05.474ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38142025-04-30T06:43:05.474ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38152025-04-30T06:43:05.474ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38162025-04-30T06:43:05.474ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38172025-04-30T06:43:05.475ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38182025-04-30T06:43:05.479ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38192025-04-30T06:43:05.484ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38202025-04-30T06:43:05.484ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38212025-04-30T06:43:05.484ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38222025-04-30T06:43:05.485ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38232025-04-30T06:43:05.486ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
38242025-04-30T06:43:05.489ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
38252025-04-30T06:43:05.490ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38262025-04-30T06:43:05.491ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 86 6e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .n..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54289 req_id = d97aef1f-d998-4138-9a89-3716f9d7a33b
38272025-04-30T06:43:05.491ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34414,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54289 req_id = d97aef1f-d998-4138-9a89-3716f9d7a33b
38282025-04-30T06:43:05.491ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
38292025-04-30T06:43:05.491ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38302025-04-30T06:43:05.492ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54289 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d97aef1f-d998-4138-9a89-3716f9d7a33b
38312025-04-30T06:43:05.492ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4250, tv_nsec: 291341030 } })
38322025-04-30T06:43:05.492ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
38332025-04-30T06:43:05.493ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 50 6a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Pj..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50694 req_id = 2fd5448c-cdfc-4bce-a91f-cd9a15814006
38342025-04-30T06:43:05.493ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20586,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50694 req_id = 2fd5448c-cdfc-4bce-a91f-cd9a15814006
38352025-04-30T06:43:05.494ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38362025-04-30T06:43:05.494ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50694 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2fd5448c-cdfc-4bce-a91f-cd9a15814006
38372025-04-30T06:43:05.599ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38382025-04-30T06:43:05.600ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38392025-04-30T06:43:05.600ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38402025-04-30T06:43:05.600ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38412025-04-30T06:43:05.600ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
38422025-04-30T06:43:05.601ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38432025-04-30T06:43:05.602ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ca 5c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .\\..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60816 req_id = 549e56d2-88b0-4aaa-98d4-668421ac7904
38442025-04-30T06:43:05.602ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51804,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60816 req_id = 549e56d2-88b0-4aaa-98d4-668421ac7904
38452025-04-30T06:43:05.602ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
38462025-04-30T06:43:05.603ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51804, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60816 req_id = 549e56d2-88b0-4aaa-98d4-668421ac7904
38472025-04-30T06:43:05.604ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
38482025-04-30T06:43:05.604ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
38492025-04-30T06:43:05.604ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
38502025-04-30T06:43:05.610ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38512025-04-30T06:43:05.626ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38522025-04-30T06:43:05.626ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38532025-04-30T06:43:05.626ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
38542025-04-30T06:43:05.627ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38552025-04-30T06:43:05.654ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
38562025-04-30T06:43:05.656ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38572025-04-30T06:43:05.656ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38582025-04-30T06:43:05.656ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
38592025-04-30T06:43:05.658ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38602025-04-30T06:43:05.659ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 196f1c3c-96c9-4fd4-a9e4-47aecef95e71 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38612025-04-30T06:43:05.660ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 196f1c3c-96c9-4fd4-a9e4-47aecef95e71 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38622025-04-30T06:43:05.660ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 196f1c3c-96c9-4fd4-a9e4-47aecef95e71 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38632025-04-30T06:43:05.661ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 196f1c3c-96c9-4fd4-a9e4-47aecef95e71 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38642025-04-30T06:43:05.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9132c7f2-1337-4d31-a368-db56f78507da", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38652025-04-30T06:43:05.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
38662025-04-30T06:43:05.693ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 33783 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 196f1c3c-96c9-4fd4-a9e4-47aecef95e71 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38672025-04-30T06:43:05.694ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "196f1c3c-96c9-4fd4-a9e4-47aecef95e71", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38682025-04-30T06:43:05.694ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38692025-04-30T06:43:05.695ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 986e021c-b90d-4ad4-86b7-ce132ae850ad uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38702025-04-30T06:43:05.695ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 986e021c-b90d-4ad4-86b7-ce132ae850ad resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38712025-04-30T06:43:05.695ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 986e021c-b90d-4ad4-86b7-ce132ae850ad roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38722025-04-30T06:43:05.696ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 986e021c-b90d-4ad4-86b7-ce132ae850ad resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38732025-04-30T06:43:05.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "056dd5c3-af19-4b93-baaa-3257bb6bec82", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38742025-04-30T06:43:05.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
38752025-04-30T06:43:05.702ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 7159 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 986e021c-b90d-4ad4-86b7-ce132ae850ad response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
38762025-04-30T06:43:05.703ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "986e021c-b90d-4ad4-86b7-ce132ae850ad", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38772025-04-30T06:43:05.703ZDEBGtest_snapshot_unwind (oximeter-agent): registered new metric producer address = [::1]:53117 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38782025-04-30T06:43:05.787ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38792025-04-30T06:43:05.798ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
38802025-04-30T06:43:05.798ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
38812025-04-30T06:43:05.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b2e011-f740-41fd-9ff2-480e4cd7d6dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38822025-04-30T06:43:05.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
38832025-04-30T06:43:05.807ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a4bec6b-6c12-4ec1-94f1-cb5e16f789be", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38842025-04-30T06:43:05.813ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
38852025-04-30T06:43:05.820ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e361e283-84ae-4ef1-bb6a-6cff3da273a7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38862025-04-30T06:43:05.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
38872025-04-30T06:43:05.862ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d32ecb-fc90-4348-b5fe-ca983689ff62", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38882025-04-30T06:43:05.862ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
38892025-04-30T06:43:05.870ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
38902025-04-30T06:43:05.873ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7737dd37-7b9b-4772-b896-4fc2bdfde92a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38912025-04-30T06:43:05.873ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
38922025-04-30T06:43:05.878ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "124c231f-1bd9-42df-8c96-6bef0950bd0e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38932025-04-30T06:43:05.883ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
38942025-04-30T06:43:05.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e78d87d-d9ae-4a98-abe9-8f1dcdf43527", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38952025-04-30T06:43:05.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
38962025-04-30T06:43:05.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eebcae5-676b-4408-abce-92f7c7d31dbb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38972025-04-30T06:43:05.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
38982025-04-30T06:43:05.891ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fb1915e-e14f-40d4-b356-6c8c810dedfc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
38992025-04-30T06:43:05.892ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
39002025-04-30T06:43:05.893ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a239b1f-f644-4359-ac45-66aa8701806e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39012025-04-30T06:43:05.893ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
39022025-04-30T06:43:05.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3820d0e-ee48-4533-8fee-d854f0a3b2cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39032025-04-30T06:43:05.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
39042025-04-30T06:43:05.900ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3703acf4-96ff-4ec2-8d74-c1842de3e3d3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39052025-04-30T06:43:05.901ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
39062025-04-30T06:43:05.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "683ce36a-7866-4afc-94e7-50c27f78394b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39072025-04-30T06:43:05.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
39082025-04-30T06:43:05.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a300c9-94aa-4222-8cda-cacf34a70809", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39092025-04-30T06:43:05.907ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
39102025-04-30T06:43:05.910ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0cdc602-d3c0-4635-8d8b-3149a5292b72", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39112025-04-30T06:43:05.910ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
39122025-04-30T06:43:05.912ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87ae7457-d8fb-4043-835b-c0a73fb2fcf3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39132025-04-30T06:43:05.912ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
39142025-04-30T06:43:05.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b93b0394-dabd-4037-9587-3e4697096416", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39152025-04-30T06:43:05.918ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
39162025-04-30T06:43:05.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7606dc8-4516-46ce-bf11-17f2376a7a30", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39172025-04-30T06:43:05.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
39182025-04-30T06:43:05.921ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39192025-04-30T06:43:05.922ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39202025-04-30T06:43:05.922ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39212025-04-30T06:43:05.922ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
39222025-04-30T06:43:05.923ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39232025-04-30T06:43:05.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff27a59-306a-45e6-aad4-1e89be1b5b19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39242025-04-30T06:43:05.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
39252025-04-30T06:43:05.927ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6beaee79-8c2f-4b08-be9f-6d6e4f948309", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39262025-04-30T06:43:05.927ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
39272025-04-30T06:43:05.928ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b40809eb-db9c-434a-9120-c1ffbaf48236", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39282025-04-30T06:43:05.929ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
39292025-04-30T06:43:05.930ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6499455b-f06d-4536-8436-777e470483ef", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39302025-04-30T06:43:05.931ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
39312025-04-30T06:43:05.931ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39322025-04-30T06:43:05.937ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39332025-04-30T06:43:05.937ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39342025-04-30T06:43:05.937ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
39352025-04-30T06:43:05.938ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39362025-04-30T06:43:05.942ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39372025-04-30T06:43:05.942ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
39382025-04-30T06:43:05.942ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39392025-04-30T06:43:05.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75671e4e-6d85-446c-8d58-e5fd6e0e3fd2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39402025-04-30T06:43:05.945ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
39412025-04-30T06:43:05.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a447589-dee9-4abe-9ac8-0cd2abf061e9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39422025-04-30T06:43:05.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
39432025-04-30T06:43:05.963ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b493fce2-1446-4b3a-9582-1cc7cbfc4a30", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39442025-04-30T06:43:05.964ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
39452025-04-30T06:43:05.967ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51aa0349-e216-4a31-a9b7-f0a70a46721a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39462025-04-30T06:43:05.967ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
39472025-04-30T06:43:05.976ZTRACtest_snapshot_unwind (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39482025-04-30T06:43:05.976ZTRACtest_snapshot_unwind (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39492025-04-30T06:43:05.978ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfd64733-c654-4133-89fb-0a8ff4193c45", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39502025-04-30T06:43:05.978ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
39512025-04-30T06:43:05.982ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6b39e65-496d-44e9-950d-7909a68f0a6f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39522025-04-30T06:43:05.983ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
39532025-04-30T06:43:05.984ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86c80e95-a676-4485-83f0-0dc67e172c7a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39542025-04-30T06:43:05.984ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
39552025-04-30T06:43:05.985ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38cd19e6-921e-4a5b-936f-05e9a1dd2dda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39562025-04-30T06:43:05.986ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
39572025-04-30T06:43:06.008ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1431f54e-a758-47e8-b858-41fdb39245ec", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39582025-04-30T06:43:06.009ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
39592025-04-30T06:43:06.015ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
39602025-04-30T06:43:06.016ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f056e279-07f4-42e5-9487-3d56651204f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39612025-04-30T06:43:06.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
39622025-04-30T06:43:06.018ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47a54583-d36b-4251-bd4e-82afe56b1367", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39632025-04-30T06:43:06.018ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
39642025-04-30T06:43:06.020ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b20ca15f-c0e3-4b34-9c9f-ee23437e4232", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39652025-04-30T06:43:06.020ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
39662025-04-30T06:43:06.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c779726a-d073-4b0d-996e-373c5866295c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39672025-04-30T06:43:06.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
39682025-04-30T06:43:06.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec94909-be31-4335-919e-693423e2b3c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39692025-04-30T06:43:06.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
39702025-04-30T06:43:06.027ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "321bb3d0-1b58-4f50-a4c7-27135911b462", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39712025-04-30T06:43:06.027ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
39722025-04-30T06:43:06.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af1b9f72-dc69-4d6b-9333-f198fbaf53e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39732025-04-30T06:43:06.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
39742025-04-30T06:43:06.032ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c71794f8-2502-4b01-9e9a-8402b54175fa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39752025-04-30T06:43:06.032ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
39762025-04-30T06:43:06.033ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ccacc60-65af-4e41-91ae-f205bf3b2eb2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39772025-04-30T06:43:06.033ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
39782025-04-30T06:43:06.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4162f1d8-f8f0-4281-8b9b-d629e5b3a2b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39792025-04-30T06:43:06.035ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
39802025-04-30T06:43:06.035ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7629a7d-9721-4d65-9f4c-462b3c73e765", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39812025-04-30T06:43:06.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
39822025-04-30T06:43:06.037ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bdd2725-1485-429c-9d32-f8a9fb7dc602", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
39832025-04-30T06:43:06.037ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39842025-04-30T06:43:06.037ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39852025-04-30T06:43:06.037ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39862025-04-30T06:43:06.037ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39872025-04-30T06:43:06.038ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39882025-04-30T06:43:06.059ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39892025-04-30T06:43:06.063ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39902025-04-30T06:43:06.063ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39912025-04-30T06:43:06.063ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39922025-04-30T06:43:06.063ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39932025-04-30T06:43:06.063ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39942025-04-30T06:43:06.063ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39952025-04-30T06:43:06.069ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39962025-04-30T06:43:06.069ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39972025-04-30T06:43:06.069ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39982025-04-30T06:43:06.069ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
39992025-04-30T06:43:06.071ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40002025-04-30T06:43:06.096ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40012025-04-30T06:43:06.116ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40022025-04-30T06:43:06.116ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40032025-04-30T06:43:06.116ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40042025-04-30T06:43:06.116ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40052025-04-30T06:43:06.219ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
40062025-04-30T06:43:06.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
40072025-04-30T06:43:06.220ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40082025-04-30T06:43:06.226ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40092025-04-30T06:43:06.226ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40102025-04-30T06:43:06.226ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
40112025-04-30T06:43:06.227ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40122025-04-30T06:43:06.227ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:53260 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40132025-04-30T06:43:06.228ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:53260 req_id = 80e7dec6-094a-462b-88e1-41487597f5b8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40142025-04-30T06:43:06.229ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 827 local_addr = [::1]:45714 method = GET remote_addr = [::1]:53260 req_id = 80e7dec6-094a-462b-88e1-41487597f5b8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40152025-04-30T06:43:06.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "80e7dec6-094a-462b-88e1-41487597f5b8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40162025-04-30T06:43:06.232ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40172025-04-30T06:43:06.232ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40182025-04-30T06:43:06.232ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
40192025-04-30T06:43:06.234ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40202025-04-30T06:43:06.234ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40212025-04-30T06:43:06.235ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40222025-04-30T06:43:06.235ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40232025-04-30T06:43:06.235ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40242025-04-30T06:43:06.258ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40252025-04-30T06:43:06.259ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
40262025-04-30T06:43:06.260ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40272025-04-30T06:43:06.269ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40282025-04-30T06:43:06.283ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40292025-04-30T06:43:06.283ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40302025-04-30T06:43:06.283ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40312025-04-30T06:43:06.285ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40322025-04-30T06:43:06.392ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40332025-04-30T06:43:06.392ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40342025-04-30T06:43:06.393ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 918.945989ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40352025-04-30T06:43:06.393ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40362025-04-30T06:43:06.393ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40372025-04-30T06:43:06.393ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40382025-04-30T06:43:06.393ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40392025-04-30T06:43:06.393ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40402025-04-30T06:43:06.394ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40412025-04-30T06:43:06.399ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40422025-04-30T06:43:06.404ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40432025-04-30T06:43:06.404ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40442025-04-30T06:43:06.404ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40452025-04-30T06:43:06.404ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40462025-04-30T06:43:06.407ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
40472025-04-30T06:43:06.407ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40482025-04-30T06:43:06.408ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d8 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54521 req_id = c7bb3a27-bb56-4e7c-a39b-91774cdd0dae
40492025-04-30T06:43:06.409ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55370,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54521 req_id = c7bb3a27-bb56-4e7c-a39b-91774cdd0dae
40502025-04-30T06:43:06.409ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
40512025-04-30T06:43:06.409ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40522025-04-30T06:43:06.409ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54521 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c7bb3a27-bb56-4e7c-a39b-91774cdd0dae
40532025-04-30T06:43:06.411ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4251, tv_nsec: 210071129 } })
40542025-04-30T06:43:06.411ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
40552025-04-30T06:43:06.411ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5e 3c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ^<..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62631 req_id = dade03e3-bd69-4c6c-9374-ebd285fa35a6
40562025-04-30T06:43:06.411ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24124,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62631 req_id = dade03e3-bd69-4c6c-9374-ebd285fa35a6
40572025-04-30T06:43:06.411ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40582025-04-30T06:43:06.411ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62631 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dade03e3-bd69-4c6c-9374-ebd285fa35a6
40592025-04-30T06:43:06.457ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40602025-04-30T06:43:06.459ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ea ab 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58806 req_id = 60f58c0d-e25e-4883-b6fe-a912f8c95d2c
40612025-04-30T06:43:06.459ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60075,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58806 req_id = 60f58c0d-e25e-4883-b6fe-a912f8c95d2c
40622025-04-30T06:43:06.459ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
40632025-04-30T06:43:06.459ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60075, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58806 req_id = 60f58c0d-e25e-4883-b6fe-a912f8c95d2c
40642025-04-30T06:43:06.460ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
40652025-04-30T06:43:06.460ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
40662025-04-30T06:43:06.460ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
40672025-04-30T06:43:06.461ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
40682025-04-30T06:43:06.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1cb168e-feb5-4881-81e7-e933d1122c63", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40692025-04-30T06:43:06.508ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
40702025-04-30T06:43:06.509ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fea12437-a27d-4443-b408-43dc27f4ff44", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40712025-04-30T06:43:06.509ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
40722025-04-30T06:43:06.512ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40732025-04-30T06:43:06.512ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40742025-04-30T06:43:06.512ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40752025-04-30T06:43:06.512ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
40762025-04-30T06:43:06.513ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40772025-04-30T06:43:06.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b8b3c94-9087-4f05-b62f-6eb03ebdcd22", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40782025-04-30T06:43:06.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
40792025-04-30T06:43:06.516ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40802025-04-30T06:43:06.524ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40812025-04-30T06:43:06.524ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40822025-04-30T06:43:06.524ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
40832025-04-30T06:43:06.525ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40842025-04-30T06:43:06.527ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da2b859f-00e8-4f60-9d2e-0c52cb599b2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40852025-04-30T06:43:06.527ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
40862025-04-30T06:43:06.528ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40872025-04-30T06:43:06.528ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
40882025-04-30T06:43:06.529ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40892025-04-30T06:43:06.564ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96f53e4-3b15-4a2f-8351-67e7cb5ba386", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40902025-04-30T06:43:06.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
40912025-04-30T06:43:06.601ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aac759d-8c02-4f20-85e7-0eb9d71a9560", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40922025-04-30T06:43:06.601ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
40932025-04-30T06:43:06.614ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcc31084-e0a9-4a71-8320-353866525f3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40942025-04-30T06:43:06.614ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
40952025-04-30T06:43:06.618ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95f8c1d8-c74e-4f24-aa39-f49794523015", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:05 GMT"} })
40962025-04-30T06:43:06.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
40972025-04-30T06:43:06.635ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "965fc09e-9f08-44fa-b816-273679e73371", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
40982025-04-30T06:43:06.635ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
40992025-04-30T06:43:06.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67669956-4eef-4472-a429-65e116d88130", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41002025-04-30T06:43:06.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
41012025-04-30T06:43:06.641ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61bf5771-10ae-4316-8865-96cdaad1f8b6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41022025-04-30T06:43:06.641ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
41032025-04-30T06:43:06.648ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9d58bfd-b83a-45e7-92a8-9b5d55ba4770", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41042025-04-30T06:43:06.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
41052025-04-30T06:43:06.649ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
41062025-04-30T06:43:06.652ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bd7dcb1-e5d1-458e-a776-5481c6019c98", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41072025-04-30T06:43:06.652ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
41082025-04-30T06:43:06.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b91bde75-b3d0-4abf-ad2e-621023237b90", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41092025-04-30T06:43:06.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
41102025-04-30T06:43:06.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "043849df-96c4-4f82-9fd6-6fc688a512f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41112025-04-30T06:43:06.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
41122025-04-30T06:43:06.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b5ec327-ea9d-445f-b601-e52afe7021cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41132025-04-30T06:43:06.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
41142025-04-30T06:43:06.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53394404-1cf7-4bf8-b500-ed3668910cc2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41152025-04-30T06:43:06.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
41162025-04-30T06:43:06.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87836c02-1f06-4b64-86f5-b7bcf88eb60b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41172025-04-30T06:43:06.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
41182025-04-30T06:43:06.679ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3d4801a-2e27-4c18-a05d-a1b4179e5d32", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41192025-04-30T06:43:06.679ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
41202025-04-30T06:43:06.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51ffeae-ce21-498f-b97a-b92e3c75d9c0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41212025-04-30T06:43:06.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
41222025-04-30T06:43:06.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5df2d0f4-32ec-4c1a-a2b8-858a5040a5b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41232025-04-30T06:43:06.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
41242025-04-30T06:43:06.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f68bcb67-90a4-4c88-8b46-fedfad939f06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41252025-04-30T06:43:06.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
41262025-04-30T06:43:06.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04f5300d-de1e-499f-9ca6-eade005b0639", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41272025-04-30T06:43:06.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
41282025-04-30T06:43:06.695ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c31fcdb7-b46f-4b89-a3ef-27f1ef44551f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41292025-04-30T06:43:06.696ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
41302025-04-30T06:43:06.700ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41312025-04-30T06:43:06.700ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41322025-04-30T06:43:06.700ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41332025-04-30T06:43:06.701ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
41342025-04-30T06:43:06.702ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41352025-04-30T06:43:06.704ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41362025-04-30T06:43:06.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41372025-04-30T06:43:06.708ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41382025-04-30T06:43:06.741ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41392025-04-30T06:43:06.741ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41402025-04-30T06:43:06.741ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
41412025-04-30T06:43:06.742ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41422025-04-30T06:43:06.743ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41432025-04-30T06:43:06.745ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41442025-04-30T06:43:06.745ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41452025-04-30T06:43:06.745ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41462025-04-30T06:43:06.748ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41472025-04-30T06:43:06.749ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
41482025-04-30T06:43:06.750ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41492025-04-30T06:43:06.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d54ad36-7ec3-491a-8fca-ccb5dfd77c63", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41502025-04-30T06:43:06.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
41512025-04-30T06:43:06.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9daf395-37df-4c71-9cde-d88e75d064b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41522025-04-30T06:43:06.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
41532025-04-30T06:43:06.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71764700-5bbb-4ffe-a426-2bb6a273d4a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41542025-04-30T06:43:06.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
41552025-04-30T06:43:06.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3582b7d2-a9b5-401f-b4e5-e06a0075e8f3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41562025-04-30T06:43:06.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
41572025-04-30T06:43:06.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99b742e6-307c-4193-a696-787c33bbfb38", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41582025-04-30T06:43:06.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
41592025-04-30T06:43:06.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a0a013c-7c6f-44dc-92a3-56ec4f009949", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41602025-04-30T06:43:06.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
41612025-04-30T06:43:06.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ee1d09-c9e1-466e-b3fe-940dc0420222", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41622025-04-30T06:43:06.841ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
41632025-04-30T06:43:06.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5acc998b-965c-454f-8042-acecab8a5214", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41642025-04-30T06:43:06.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
41652025-04-30T06:43:06.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d0b5ae7-f69c-4d2b-875d-19816b7d9881", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41662025-04-30T06:43:06.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
41672025-04-30T06:43:06.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7443e5c-11a6-46e3-b881-b3563076aaa0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41682025-04-30T06:43:06.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
41692025-04-30T06:43:06.878ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
41702025-04-30T06:43:06.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "945ebcc7-6eaf-406f-97ee-5dd0fd84cc43", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41712025-04-30T06:43:06.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
41722025-04-30T06:43:06.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9932cdfa-93a7-4ea4-8eff-dd78ca7d7be2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41732025-04-30T06:43:06.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
41742025-04-30T06:43:06.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73cdba3a-6e97-4546-8142-f7d01e581678", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41752025-04-30T06:43:06.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
41762025-04-30T06:43:06.897ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "051df561-5524-4c8c-b6e4-d2f8e141dd0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41772025-04-30T06:43:06.898ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
41782025-04-30T06:43:06.902ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e14271-23a8-4517-9f3f-ac2ab3ed75e3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41792025-04-30T06:43:06.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
41802025-04-30T06:43:06.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c840c09-f080-497e-b61a-9fcce4cd055c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41812025-04-30T06:43:06.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
41822025-04-30T06:43:06.910ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7444960e-e434-4e78-8b10-49e5b9137c6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41832025-04-30T06:43:06.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
41842025-04-30T06:43:06.912ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "830ebb78-c5a7-4de6-80c5-8e55d26de4af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41852025-04-30T06:43:06.912ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
41862025-04-30T06:43:06.913ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "142d9d18-d5f5-4d46-b31e-dfeba07a5b22", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41872025-04-30T06:43:06.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
41882025-04-30T06:43:06.915ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fad0fbc1-0dbc-4da0-a9ef-80d21356443b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41892025-04-30T06:43:06.915ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
41902025-04-30T06:43:06.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02605bfe-5732-4efb-8e15-f3291577cda1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
41912025-04-30T06:43:06.917ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
41922025-04-30T06:43:06.917ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41932025-04-30T06:43:06.917ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41942025-04-30T06:43:06.917ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41952025-04-30T06:43:06.918ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41962025-04-30T06:43:06.931ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41972025-04-30T06:43:06.931ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41982025-04-30T06:43:06.931ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41992025-04-30T06:43:06.931ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
42002025-04-30T06:43:06.932ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42012025-04-30T06:43:06.933ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42022025-04-30T06:43:06.937ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42032025-04-30T06:43:06.937ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42042025-04-30T06:43:06.938ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42052025-04-30T06:43:06.938ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42062025-04-30T06:43:06.938ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42072025-04-30T06:43:06.939ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42082025-04-30T06:43:06.943ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42092025-04-30T06:43:06.966ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42102025-04-30T06:43:06.966ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42112025-04-30T06:43:06.966ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
42122025-04-30T06:43:06.967ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42132025-04-30T06:43:06.969ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42142025-04-30T06:43:06.972ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42152025-04-30T06:43:06.972ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42162025-04-30T06:43:06.972ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42172025-04-30T06:43:06.973ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42182025-04-30T06:43:07.011ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42192025-04-30T06:43:07.011ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
42202025-04-30T06:43:07.012ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42212025-04-30T06:43:07.012ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42222025-04-30T06:43:07.012ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42232025-04-30T06:43:07.012ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
42242025-04-30T06:43:07.013ZDEBGtest_snapshot_unwind (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
42252025-04-30T06:43:07.013ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
42262025-04-30T06:43:07.023ZTRACtest_snapshot_unwind (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
42272025-04-30T06:43:07.023ZDEBGtest_snapshot_unwind (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:07.013431246Z } timeseries_name = integration_target:integration_metric
42282025-04-30T06:43:07.025ZDEBGtest_snapshot_unwind (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
42292025-04-30T06:43:07.025ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
42302025-04-30T06:43:07.088ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
42312025-04-30T06:43:07.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
42322025-04-30T06:43:07.089ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:39610 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42332025-04-30T06:43:07.094ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42342025-04-30T06:43:07.095ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:39610 req_id = 9f5bd7d7-bfdf-4e34-85f3-a4e0c765dbc6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42352025-04-30T06:43:07.096ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1235 local_addr = [::1]:45714 method = GET remote_addr = [::1]:39610 req_id = 9f5bd7d7-bfdf-4e34-85f3-a4e0c765dbc6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42362025-04-30T06:43:07.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f5bd7d7-bfdf-4e34-85f3-a4e0c765dbc6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
42372025-04-30T06:43:07.109ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42382025-04-30T06:43:07.109ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42392025-04-30T06:43:07.109ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
42402025-04-30T06:43:07.110ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42412025-04-30T06:43:07.110ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42422025-04-30T06:43:07.110ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42432025-04-30T06:43:07.110ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42442025-04-30T06:43:07.111ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42452025-04-30T06:43:07.112ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
42462025-04-30T06:43:07.112ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42472025-04-30T06:43:07.114ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42482025-04-30T06:43:07.115ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42492025-04-30T06:43:07.118ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42502025-04-30T06:43:07.118ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42512025-04-30T06:43:07.118ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42522025-04-30T06:43:07.119ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42532025-04-30T06:43:07.120ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
42542025-04-30T06:43:07.120ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42552025-04-30T06:43:07.149ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
42562025-04-30T06:43:07.158ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42572025-04-30T06:43:07.159ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42582025-04-30T06:43:07.161ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 767.788357ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42592025-04-30T06:43:07.161ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42602025-04-30T06:43:07.162ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42612025-04-30T06:43:07.162ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42622025-04-30T06:43:07.162ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42632025-04-30T06:43:07.162ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42642025-04-30T06:43:07.163ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42652025-04-30T06:43:07.167ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42662025-04-30T06:43:07.176ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42672025-04-30T06:43:07.176ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42682025-04-30T06:43:07.176ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42692025-04-30T06:43:07.177ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42702025-04-30T06:43:07.181ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
42712025-04-30T06:43:07.181ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42722025-04-30T06:43:07.182ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 47 b8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 G............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53638 req_id = e7faa66e-8b23-45f2-ae8f-27a63c440936
42732025-04-30T06:43:07.186ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18360,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53638 req_id = e7faa66e-8b23-45f2-ae8f-27a63c440936
42742025-04-30T06:43:07.186ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
42752025-04-30T06:43:07.186ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42762025-04-30T06:43:07.187ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53638 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e7faa66e-8b23-45f2-ae8f-27a63c440936
42772025-04-30T06:43:07.187ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4251, tv_nsec: 986457027 } })
42782025-04-30T06:43:07.187ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
42792025-04-30T06:43:07.188ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ee 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49392 req_id = 207c6b25-158f-4fc9-aa16-d790a2b164cd
42802025-04-30T06:43:07.188ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49392 req_id = 207c6b25-158f-4fc9-aa16-d790a2b164cd
42812025-04-30T06:43:07.188ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42822025-04-30T06:43:07.188ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49392 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 207c6b25-158f-4fc9-aa16-d790a2b164cd
42832025-04-30T06:43:07.250ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42842025-04-30T06:43:07.251ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42852025-04-30T06:43:07.251ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42862025-04-30T06:43:07.251ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42872025-04-30T06:43:07.251ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
42882025-04-30T06:43:07.252ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42892025-04-30T06:43:07.254ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 01 db 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63601 req_id = bedb50c3-2981-4c46-b665-aafa8c50e185
42902025-04-30T06:43:07.254ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 475,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63601 req_id = bedb50c3-2981-4c46-b665-aafa8c50e185
42912025-04-30T06:43:07.255ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
42922025-04-30T06:43:07.255ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 475, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63601 req_id = bedb50c3-2981-4c46-b665-aafa8c50e185
42932025-04-30T06:43:07.256ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
42942025-04-30T06:43:07.256ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
42952025-04-30T06:43:07.256ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
42962025-04-30T06:43:07.257ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42972025-04-30T06:43:07.268ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42982025-04-30T06:43:07.268ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42992025-04-30T06:43:07.268ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
43002025-04-30T06:43:07.268ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43012025-04-30T06:43:07.279ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43022025-04-30T06:43:07.279ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
43032025-04-30T06:43:07.279ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43042025-04-30T06:43:07.280ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "175aa495-3684-4fa0-9346-8d5f17bbfb99", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43052025-04-30T06:43:07.280ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
43062025-04-30T06:43:07.289ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fb85120-0f94-4586-acb3-825a41f1bb64", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43072025-04-30T06:43:07.293ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
43082025-04-30T06:43:07.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aa2d02b-a7dd-458b-a2ef-6b58103531f0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43092025-04-30T06:43:07.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
43102025-04-30T06:43:07.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6dd3190-9021-44ca-99c8-4fdb36bc4ff9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43112025-04-30T06:43:07.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
43122025-04-30T06:43:07.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50eee2b6-d681-4bc4-b5e0-eb427465aeb9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43132025-04-30T06:43:07.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
43142025-04-30T06:43:07.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba7d8380-deb6-464a-ae6e-3c92e986d7fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43152025-04-30T06:43:07.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
43162025-04-30T06:43:07.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db8fedf6-82ec-44e6-a2a8-bee7c00f95c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43172025-04-30T06:43:07.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
43182025-04-30T06:43:07.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cad0fd39-533d-4b18-8dda-bf114c411a3c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43192025-04-30T06:43:07.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
43202025-04-30T06:43:07.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7132de90-336b-4037-977c-6a2eb141e03e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43212025-04-30T06:43:07.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
43222025-04-30T06:43:07.336ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
43232025-04-30T06:43:07.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71075d98-31b2-4fe1-9c3b-4f61895d2bb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43242025-04-30T06:43:07.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
43252025-04-30T06:43:07.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10bb322d-23f3-4aa1-b4d5-d45d40eea010", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43262025-04-30T06:43:07.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
43272025-04-30T06:43:07.341ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5968b38-9149-4446-8eda-736c4fd486de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43282025-04-30T06:43:07.343ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
43292025-04-30T06:43:07.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7656011f-05c8-45d0-b6b4-6910966e0b15", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43302025-04-30T06:43:07.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
43312025-04-30T06:43:07.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8c93db0-474d-4dab-a115-84fab73f5520", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43322025-04-30T06:43:07.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
43332025-04-30T06:43:07.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8176f7e7-6af5-49ae-b6d7-40466fd7394f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43342025-04-30T06:43:07.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
43352025-04-30T06:43:07.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e8d722c-3a94-4165-b8b7-d96084bba7d6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43362025-04-30T06:43:07.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
43372025-04-30T06:43:07.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cdac219-8628-4e8f-b3ed-2d02813a762d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43382025-04-30T06:43:07.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
43392025-04-30T06:43:07.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82c3f4d4-d917-407d-99cc-496d2531c4c8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43402025-04-30T06:43:07.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
43412025-04-30T06:43:07.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "102e6148-f098-4dcb-afd5-1f46c9c73dde", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43422025-04-30T06:43:07.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
43432025-04-30T06:43:07.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a11ee2-7aab-4458-ad14-f8d5405054a7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43442025-04-30T06:43:07.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
43452025-04-30T06:43:07.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f56440-8013-416f-a1e9-fe13b9262379", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43462025-04-30T06:43:07.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
43472025-04-30T06:43:07.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a2d63c-14db-4910-ba4e-4d45a7254237", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43482025-04-30T06:43:07.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
43492025-04-30T06:43:07.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb8146a-8363-4d88-9936-6eb772b721b3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43502025-04-30T06:43:07.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
43512025-04-30T06:43:07.374ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "536234de-66d5-4b21-9457-e94edbc6f0fc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43522025-04-30T06:43:07.374ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
43532025-04-30T06:43:07.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b44c2db3-3ce7-440f-baa7-235821d8f79d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43542025-04-30T06:43:07.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
43552025-04-30T06:43:07.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd4ab565-ed70-4a5b-8f76-7a7056781301", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43562025-04-30T06:43:07.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
43572025-04-30T06:43:07.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fcf5696-1cf2-4c89-ad08-4d3f2b80a2be", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43582025-04-30T06:43:07.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
43592025-04-30T06:43:07.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "720270f6-a5f3-41bb-97a9-8c67f6b43974", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43602025-04-30T06:43:07.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
43612025-04-30T06:43:07.387ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43622025-04-30T06:43:07.387ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43632025-04-30T06:43:07.388ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43642025-04-30T06:43:07.388ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
43652025-04-30T06:43:07.389ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43662025-04-30T06:43:07.392ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43672025-04-30T06:43:07.396ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43682025-04-30T06:43:07.396ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43692025-04-30T06:43:07.396ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
43702025-04-30T06:43:07.397ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43712025-04-30T06:43:07.397ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55d6ec56-0502-403a-9031-57368451dc8f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43722025-04-30T06:43:07.397ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
43732025-04-30T06:43:07.403ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43742025-04-30T06:43:07.403ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
43752025-04-30T06:43:07.404ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43762025-04-30T06:43:07.407ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5844a74-ae22-4e35-912e-33dd9108ebf7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43772025-04-30T06:43:07.407ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
43782025-04-30T06:43:07.409ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9451e912-a251-4e0d-a7df-5dcc5cbd9c5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43792025-04-30T06:43:07.410ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
43802025-04-30T06:43:07.416ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd49a04-b042-4348-a21b-701798255d34", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43812025-04-30T06:43:07.422ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
43822025-04-30T06:43:07.423ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a1766e1-fa5e-40d8-99ea-c6853058bdc8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43832025-04-30T06:43:07.423ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
43842025-04-30T06:43:07.424ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a3de41-4861-4f3e-9524-10795b1496d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43852025-04-30T06:43:07.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
43862025-04-30T06:43:07.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ace51dd-0124-431f-9c05-dca3b2e52bb3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43872025-04-30T06:43:07.426ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
43882025-04-30T06:43:07.428ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "636c531c-0428-4cc4-a284-170dd382df5f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43892025-04-30T06:43:07.443ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
43902025-04-30T06:43:07.446ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8351bea0-ec2e-462d-b76d-6d60fa12bfee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43912025-04-30T06:43:07.446ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
43922025-04-30T06:43:07.448ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72b1802b-2546-44b6-85d1-a305d21e9a0b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43932025-04-30T06:43:07.448ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
43942025-04-30T06:43:07.450ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e766d9b-e41c-404e-84a9-c830c3fa04aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43952025-04-30T06:43:07.450ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
43962025-04-30T06:43:07.451ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2689a413-64f6-42f5-8f64-9414524f5c78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43972025-04-30T06:43:07.452ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
43982025-04-30T06:43:07.453ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52026c27-7c53-4fcc-ae2e-f7c8a1db85dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
43992025-04-30T06:43:07.453ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
44002025-04-30T06:43:07.454ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51b5b5da-aac8-438f-a867-d1ae5adeb410", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
44012025-04-30T06:43:07.454ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
44022025-04-30T06:43:07.462ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "187b57b7-5303-46d1-b4f2-eac6c4c74637", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
44032025-04-30T06:43:07.463ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
44042025-04-30T06:43:07.465ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e75541fe-7f19-47ae-926b-f3b2f651bbcb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
44052025-04-30T06:43:07.465ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
44062025-04-30T06:43:07.467ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8afefdf7-f1d0-4894-93ce-c9045b7bdad8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
44072025-04-30T06:43:07.467ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44082025-04-30T06:43:07.467ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44092025-04-30T06:43:07.467ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44102025-04-30T06:43:07.467ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44112025-04-30T06:43:07.468ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44122025-04-30T06:43:07.475ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44132025-04-30T06:43:07.479ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44142025-04-30T06:43:07.479ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44152025-04-30T06:43:07.479ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44162025-04-30T06:43:07.479ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44172025-04-30T06:43:07.479ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44182025-04-30T06:43:07.479ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44192025-04-30T06:43:07.480ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
44202025-04-30T06:43:07.481ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44212025-04-30T06:43:07.486ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44222025-04-30T06:43:07.487ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44232025-04-30T06:43:07.487ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44242025-04-30T06:43:07.488ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44252025-04-30T06:43:07.609ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
44262025-04-30T06:43:07.609ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
44272025-04-30T06:43:07.610ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44282025-04-30T06:43:07.610ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44292025-04-30T06:43:07.610ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44302025-04-30T06:43:07.610ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
44312025-04-30T06:43:07.611ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44322025-04-30T06:43:07.611ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40708 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44332025-04-30T06:43:07.612ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40708 req_id = bba94459-7006-44ba-8744-9ac069ea426f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44342025-04-30T06:43:07.613ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 500 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40708 req_id = bba94459-7006-44ba-8744-9ac069ea426f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44352025-04-30T06:43:07.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba94459-7006-44ba-8744-9ac069ea426f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:06 GMT"} })
44362025-04-30T06:43:07.618ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44372025-04-30T06:43:07.618ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44382025-04-30T06:43:07.618ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
44392025-04-30T06:43:07.620ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44402025-04-30T06:43:07.620ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44412025-04-30T06:43:07.620ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44422025-04-30T06:43:07.620ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44432025-04-30T06:43:07.622ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44442025-04-30T06:43:07.622ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44452025-04-30T06:43:07.627ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44462025-04-30T06:43:07.627ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44472025-04-30T06:43:07.627ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
44482025-04-30T06:43:07.628ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44492025-04-30T06:43:07.653ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44502025-04-30T06:43:07.659ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44512025-04-30T06:43:07.659ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44522025-04-30T06:43:07.659ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44532025-04-30T06:43:07.660ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44542025-04-30T06:43:07.662ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
44552025-04-30T06:43:07.663ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44562025-04-30T06:43:07.666ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 842b93cd-db19-44f9-aaff-ada3cd384964 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44572025-04-30T06:43:07.666ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 842b93cd-db19-44f9-aaff-ada3cd384964 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44582025-04-30T06:43:07.666ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 842b93cd-db19-44f9-aaff-ada3cd384964 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44592025-04-30T06:43:07.667ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 842b93cd-db19-44f9-aaff-ada3cd384964 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44602025-04-30T06:43:07.667ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44612025-04-30T06:43:07.667ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
44622025-04-30T06:43:07.669ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44632025-04-30T06:43:07.672ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 5085 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 842b93cd-db19-44f9-aaff-ada3cd384964 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44642025-04-30T06:43:07.672ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "842b93cd-db19-44f9-aaff-ada3cd384964", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
44652025-04-30T06:43:07.672ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44662025-04-30T06:43:07.672ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 954259b4-31a2-470a-b8b6-44fa81774a55 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44672025-04-30T06:43:07.672ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 954259b4-31a2-470a-b8b6-44fa81774a55 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44682025-04-30T06:43:07.672ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 954259b4-31a2-470a-b8b6-44fa81774a55 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44692025-04-30T06:43:07.673ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 954259b4-31a2-470a-b8b6-44fa81774a55 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44702025-04-30T06:43:07.687ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 15136 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 954259b4-31a2-470a-b8b6-44fa81774a55 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
44712025-04-30T06:43:07.687ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "954259b4-31a2-470a-b8b6-44fa81774a55", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
44722025-04-30T06:43:07.687ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44732025-04-30T06:43:07.688ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
44742025-04-30T06:43:07.688ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
44752025-04-30T06:43:07.709ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44762025-04-30T06:43:07.709ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44772025-04-30T06:43:07.711ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44782025-04-30T06:43:07.711ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44792025-04-30T06:43:07.711ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44802025-04-30T06:43:07.711ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44812025-04-30T06:43:07.711ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44822025-04-30T06:43:07.712ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44832025-04-30T06:43:07.712ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 550.50245ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44842025-04-30T06:43:07.712ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44852025-04-30T06:43:07.712ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44862025-04-30T06:43:07.712ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44872025-04-30T06:43:07.712ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44882025-04-30T06:43:07.712ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44892025-04-30T06:43:07.712ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44902025-04-30T06:43:07.752ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44912025-04-30T06:43:07.758ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44922025-04-30T06:43:07.758ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44932025-04-30T06:43:07.758ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44942025-04-30T06:43:07.758ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44952025-04-30T06:43:07.762ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
44962025-04-30T06:43:07.762ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44972025-04-30T06:43:07.770ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 68 5f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 h_..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65368 req_id = 979f832c-8135-4e47-832c-62c6b6b7d246
44982025-04-30T06:43:07.770ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26719,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65368 req_id = 979f832c-8135-4e47-832c-62c6b6b7d246
44992025-04-30T06:43:07.771ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
45002025-04-30T06:43:07.772ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45012025-04-30T06:43:07.772ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65368 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 979f832c-8135-4e47-832c-62c6b6b7d246
45022025-04-30T06:43:07.773ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4252, tv_nsec: 571866878 } })
45032025-04-30T06:43:07.773ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
45042025-04-30T06:43:07.773ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 c8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58280 req_id = 2d985042-f733-48cb-8f51-60f734d405ba
45052025-04-30T06:43:07.773ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9416,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58280 req_id = 2d985042-f733-48cb-8f51-60f734d405ba
45062025-04-30T06:43:07.773ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45072025-04-30T06:43:07.773ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58280 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2d985042-f733-48cb-8f51-60f734d405ba
45082025-04-30T06:43:07.859ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45092025-04-30T06:43:07.863ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d8 9b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60781 req_id = f5abbcb8-8434-401e-b0b4-b975a584b480
45102025-04-30T06:43:07.863ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55451,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60781 req_id = f5abbcb8-8434-401e-b0b4-b975a584b480
45112025-04-30T06:43:07.864ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
45122025-04-30T06:43:07.864ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55451, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60781 req_id = f5abbcb8-8434-401e-b0b4-b975a584b480
45132025-04-30T06:43:07.865ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
45142025-04-30T06:43:07.865ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
45152025-04-30T06:43:07.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
45162025-04-30T06:43:07.867ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac025332-816c-4681-aae7-138baa020a14", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45172025-04-30T06:43:07.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
45182025-04-30T06:43:07.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e04d15df-0fe3-4558-a6ed-097242c21000", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45192025-04-30T06:43:07.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
45202025-04-30T06:43:07.875ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
45212025-04-30T06:43:07.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83230315-b855-4e3f-a765-ddc2982ef955", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45222025-04-30T06:43:07.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
45232025-04-30T06:43:07.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a0c548-30a3-40d4-b68c-97c873587bcf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45242025-04-30T06:43:07.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
45252025-04-30T06:43:07.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46512744-12de-40b8-b399-7a805cfb6f77", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45262025-04-30T06:43:07.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
45272025-04-30T06:43:07.925ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45282025-04-30T06:43:07.926ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45292025-04-30T06:43:07.926ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45302025-04-30T06:43:07.926ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
45312025-04-30T06:43:07.929ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45322025-04-30T06:43:07.931ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1389c94-2b49-4ee9-8b34-6536ee54ddc3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45332025-04-30T06:43:07.931ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
45342025-04-30T06:43:07.934ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45352025-04-30T06:43:07.942ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45362025-04-30T06:43:07.942ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45372025-04-30T06:43:07.942ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
45382025-04-30T06:43:07.943ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45392025-04-30T06:43:07.946ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45402025-04-30T06:43:07.946ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
45412025-04-30T06:43:07.947ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45422025-04-30T06:43:07.947ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba849011-6dd8-4a07-97cb-907181b63631", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45432025-04-30T06:43:07.947ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
45442025-04-30T06:43:07.953ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c110c04-f112-46bb-8868-d76d93baf5d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45452025-04-30T06:43:07.953ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
45462025-04-30T06:43:07.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28bac660-b7f1-4e57-ab61-ac6ab201b84b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45472025-04-30T06:43:07.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
45482025-04-30T06:43:07.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb59e43-3e13-453a-a4b5-808aa7acf3cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45492025-04-30T06:43:07.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
45502025-04-30T06:43:07.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75718e05-1a3f-48ce-8df7-5f65d818d057", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45512025-04-30T06:43:07.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
45522025-04-30T06:43:07.962ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea5adb71-9d26-4ca8-90c7-3c3716ec5642", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45532025-04-30T06:43:07.963ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
45542025-04-30T06:43:07.964ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fb9461-1572-4c91-8ef0-b7b4cf04efde", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45552025-04-30T06:43:07.964ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
45562025-04-30T06:43:07.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "470b1fa7-6617-443d-aee6-1b76c19f3c1d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45572025-04-30T06:43:07.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
45582025-04-30T06:43:07.971ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c0209b9-602d-4716-a27d-cd80720f4c2d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45592025-04-30T06:43:07.971ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
45602025-04-30T06:43:07.981ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45612025-04-30T06:43:07.982ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45622025-04-30T06:43:07.983ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
45632025-04-30T06:43:07.985ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
45642025-04-30T06:43:07.985ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
45652025-04-30T06:43:07.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a1dfd3a-d2a9-4ed5-a0cd-d82c6565c8c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45662025-04-30T06:43:07.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
45672025-04-30T06:43:07.990ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
45682025-04-30T06:43:07.990ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
45692025-04-30T06:43:07.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58044af2-394c-4ec7-bbd6-4e95cf9bf5ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45702025-04-30T06:43:07.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
45712025-04-30T06:43:07.993ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
45722025-04-30T06:43:07.993ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45732025-04-30T06:43:07.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "871efc89-b194-4740-86e4-0c103f470e4a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45742025-04-30T06:43:07.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
45752025-04-30T06:43:07.998ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "212d9b2c-dd38-4959-8e78-aab23fc505fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45762025-04-30T06:43:07.998ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
45772025-04-30T06:43:07.999ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "405170a5-645a-4a6a-aa85-2188cd5c0daf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45782025-04-30T06:43:08.001ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
45792025-04-30T06:43:08.001ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c917a447-81d0-4358-ac29-f237ca21e37d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45802025-04-30T06:43:08.002ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
45812025-04-30T06:43:08.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df532cf3-098d-4eaa-90c0-a69aa152b959", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45822025-04-30T06:43:08.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
45832025-04-30T06:43:08.004ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a29c4e9-b462-406c-9f68-ee602d798b21", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45842025-04-30T06:43:08.004ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
45852025-04-30T06:43:08.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edab9541-9892-4d99-8056-b9702921d5d9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45862025-04-30T06:43:08.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
45872025-04-30T06:43:08.008ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "988409ab-1999-4fd6-9c38-55345a32690a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45882025-04-30T06:43:08.008ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
45892025-04-30T06:43:08.011ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4815014-dc52-46c2-815c-d0151c2d1cb5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45902025-04-30T06:43:08.011ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
45912025-04-30T06:43:08.033ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3d14775-7197-46c4-a8ca-293074cf2053", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45922025-04-30T06:43:08.033ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
45932025-04-30T06:43:08.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66c8c713-bada-4cf7-9a50-8ed8f93e3a6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
45942025-04-30T06:43:08.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
45952025-04-30T06:43:08.036ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45962025-04-30T06:43:08.037ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45972025-04-30T06:43:08.037ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45982025-04-30T06:43:08.037ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
45992025-04-30T06:43:08.037ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46002025-04-30T06:43:08.046ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46012025-04-30T06:43:08.052ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46022025-04-30T06:43:08.052ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46032025-04-30T06:43:08.052ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
46042025-04-30T06:43:08.056ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46052025-04-30T06:43:08.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0db20ae-cb8a-423c-ae3d-7506d2307c5f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46062025-04-30T06:43:08.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
46072025-04-30T06:43:08.061ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46082025-04-30T06:43:08.061ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
46092025-04-30T06:43:08.062ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46102025-04-30T06:43:08.066ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74111783-b028-4b05-8717-cf925cb7b2a0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46112025-04-30T06:43:08.066ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
46122025-04-30T06:43:08.068ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78678db8-c06f-4ce3-8825-e9db3507d7f9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46132025-04-30T06:43:08.069ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
46142025-04-30T06:43:08.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5e638ca-a6fa-4e5c-89e6-adc8d22aa877", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46152025-04-30T06:43:08.070ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
46162025-04-30T06:43:08.072ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1caea68-2c0c-4443-94f5-b4b1d58a8cf4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46172025-04-30T06:43:08.072ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
46182025-04-30T06:43:08.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a44a15-107f-475c-ad8f-62b709ae9644", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46192025-04-30T06:43:08.074ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
46202025-04-30T06:43:08.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1690a6d5-f85c-4cd9-bbbd-20608a44c4e1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46212025-04-30T06:43:08.076ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
46222025-04-30T06:43:08.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "064a1ac0-f045-41fb-bed9-6584913ac302", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46232025-04-30T06:43:08.083ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
46242025-04-30T06:43:08.084ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46252025-04-30T06:43:08.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1cb5ce-a548-4701-9921-4d0dcf255500", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46262025-04-30T06:43:08.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
46272025-04-30T06:43:08.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd18b25f-503d-43f2-8ec7-80bfadb5e98e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46282025-04-30T06:43:08.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
46292025-04-30T06:43:08.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3540ed79-b819-4486-9276-36eb9a7c37bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46302025-04-30T06:43:08.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
46312025-04-30T06:43:08.096ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68dd82e0-c4f8-4dc1-b323-5a3f370f5594", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46322025-04-30T06:43:08.096ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
46332025-04-30T06:43:08.100ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
46342025-04-30T06:43:08.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "582d1841-abf6-4bf9-884b-8cd9e91f8fdd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46352025-04-30T06:43:08.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
46362025-04-30T06:43:08.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcac6e6d-4cca-4a99-9b1b-7b3f02e48aab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46372025-04-30T06:43:08.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
46382025-04-30T06:43:08.103ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e7ad3c-0cc1-42c8-8905-995357ab5637", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46392025-04-30T06:43:08.103ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
46402025-04-30T06:43:08.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b83539-0732-47f8-91c8-020ee11ca029", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46412025-04-30T06:43:08.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
46422025-04-30T06:43:08.105ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e370056-74d2-4598-80de-9f4efff9d894", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46432025-04-30T06:43:08.105ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46442025-04-30T06:43:08.105ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46452025-04-30T06:43:08.105ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46462025-04-30T06:43:08.105ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46472025-04-30T06:43:08.106ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46482025-04-30T06:43:08.110ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46492025-04-30T06:43:08.113ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46502025-04-30T06:43:08.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46512025-04-30T06:43:08.113ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46522025-04-30T06:43:08.113ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46532025-04-30T06:43:08.113ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46542025-04-30T06:43:08.114ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46552025-04-30T06:43:08.116ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46562025-04-30T06:43:08.119ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46572025-04-30T06:43:08.119ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46582025-04-30T06:43:08.119ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46592025-04-30T06:43:08.120ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46602025-04-30T06:43:08.175ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
46612025-04-30T06:43:08.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
46622025-04-30T06:43:08.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46632025-04-30T06:43:08.175ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46642025-04-30T06:43:08.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46652025-04-30T06:43:08.175ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
46662025-04-30T06:43:08.176ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46672025-04-30T06:43:08.176ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:46840 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46682025-04-30T06:43:08.177ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:46840 req_id = 54889e88-e11a-41b0-a7fe-b2b46feeaea9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46692025-04-30T06:43:08.178ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1631 local_addr = [::1]:45714 method = GET remote_addr = [::1]:46840 req_id = 54889e88-e11a-41b0-a7fe-b2b46feeaea9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46702025-04-30T06:43:08.187ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "54889e88-e11a-41b0-a7fe-b2b46feeaea9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
46712025-04-30T06:43:08.189ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46722025-04-30T06:43:08.189ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46732025-04-30T06:43:08.189ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
46742025-04-30T06:43:08.192ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46752025-04-30T06:43:08.192ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46762025-04-30T06:43:08.192ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46772025-04-30T06:43:08.192ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46782025-04-30T06:43:08.193ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46792025-04-30T06:43:08.194ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46802025-04-30T06:43:08.198ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46812025-04-30T06:43:08.198ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46822025-04-30T06:43:08.198ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
46832025-04-30T06:43:08.199ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46842025-04-30T06:43:08.202ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46852025-04-30T06:43:08.205ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46862025-04-30T06:43:08.205ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46872025-04-30T06:43:08.205ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46882025-04-30T06:43:08.206ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46892025-04-30T06:43:08.207ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46902025-04-30T06:43:08.207ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
46912025-04-30T06:43:08.207ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46922025-04-30T06:43:08.266ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
46932025-04-30T06:43:08.266ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
46942025-04-30T06:43:08.266ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 554.283143ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
46952025-04-30T06:43:08.266ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
46962025-04-30T06:43:08.266ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46972025-04-30T06:43:08.266ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46982025-04-30T06:43:08.266ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46992025-04-30T06:43:08.266ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47002025-04-30T06:43:08.267ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47012025-04-30T06:43:08.272ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
47022025-04-30T06:43:08.277ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47032025-04-30T06:43:08.281ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47042025-04-30T06:43:08.281ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47052025-04-30T06:43:08.281ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47062025-04-30T06:43:08.282ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47072025-04-30T06:43:08.289ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
47082025-04-30T06:43:08.289ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47092025-04-30T06:43:08.294ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d9 08 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61524 req_id = 5c4e2bf1-957c-4173-8edd-ae67e7fcb66c
47102025-04-30T06:43:08.294ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55560,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61524 req_id = 5c4e2bf1-957c-4173-8edd-ae67e7fcb66c
47112025-04-30T06:43:08.294ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
47122025-04-30T06:43:08.295ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47132025-04-30T06:43:08.295ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61524 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5c4e2bf1-957c-4173-8edd-ae67e7fcb66c
47142025-04-30T06:43:08.297ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 96064598 } })
47152025-04-30T06:43:08.297ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
47162025-04-30T06:43:08.303ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1f b3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51671 req_id = ac074a34-c5f7-4b8d-ae66-72f1fbcf8a0f
47172025-04-30T06:43:08.303ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8115,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51671 req_id = ac074a34-c5f7-4b8d-ae66-72f1fbcf8a0f
47182025-04-30T06:43:08.303ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47192025-04-30T06:43:08.303ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51671 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ac074a34-c5f7-4b8d-ae66-72f1fbcf8a0f
47202025-04-30T06:43:08.354ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47212025-04-30T06:43:08.360ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47222025-04-30T06:43:08.361ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47232025-04-30T06:43:08.361ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47242025-04-30T06:43:08.361ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
47252025-04-30T06:43:08.362ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47262025-04-30T06:43:08.367ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 45 85 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c E............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61254 req_id = 57531df3-3d51-4f5c-a515-ac72453404e8
47272025-04-30T06:43:08.367ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17797,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61254 req_id = 57531df3-3d51-4f5c-a515-ac72453404e8
47282025-04-30T06:43:08.368ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
47292025-04-30T06:43:08.368ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17797, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61254 req_id = 57531df3-3d51-4f5c-a515-ac72453404e8
47302025-04-30T06:43:08.369ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
47312025-04-30T06:43:08.369ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
47322025-04-30T06:43:08.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
47332025-04-30T06:43:08.371ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47342025-04-30T06:43:08.376ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47352025-04-30T06:43:08.376ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47362025-04-30T06:43:08.376ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
47372025-04-30T06:43:08.377ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47382025-04-30T06:43:08.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e2c7265-0e4c-4e12-8780-f5f08387ce95", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47392025-04-30T06:43:08.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
47402025-04-30T06:43:08.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e6a0f6-d725-444e-b1bb-f703cbc58c53", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47412025-04-30T06:43:08.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
47422025-04-30T06:43:08.381ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47432025-04-30T06:43:08.381ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
47442025-04-30T06:43:08.382ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47452025-04-30T06:43:08.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed05596-11f4-4b98-bc74-ccf07a68cfd4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47462025-04-30T06:43:08.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
47472025-04-30T06:43:08.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa15c9c7-795b-401c-a07f-8dd5a919561e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47482025-04-30T06:43:08.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
47492025-04-30T06:43:08.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4161f85-80ab-4fc8-b76e-29a33ba3638f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47502025-04-30T06:43:08.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
47512025-04-30T06:43:08.394ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09f76ec5-b047-4c9f-93a5-a915ab3bfac9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47522025-04-30T06:43:08.395ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
47532025-04-30T06:43:08.399ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "735516b5-ee7d-4752-9700-baee81313329", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47542025-04-30T06:43:08.400ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
47552025-04-30T06:43:08.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe501e2e-75b2-47af-bef7-2f2d5da82161", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47562025-04-30T06:43:08.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
47572025-04-30T06:43:08.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3d6b1ee-6748-42d6-b21f-b954c7ed3eab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47582025-04-30T06:43:08.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
47592025-04-30T06:43:08.410ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8aa538e-4799-42ec-bcb2-f45f4ec5d6d7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47602025-04-30T06:43:08.410ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
47612025-04-30T06:43:08.414ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
47622025-04-30T06:43:08.415ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f7c4ec-d5a6-412e-b65a-fbb6dd8d67e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47632025-04-30T06:43:08.415ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
47642025-04-30T06:43:08.419ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "011f5c0b-b1aa-484b-8f2b-5f161ad07dce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47652025-04-30T06:43:08.419ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
47662025-04-30T06:43:08.421ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "088f8456-436d-4ece-915f-727de63d4274", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47672025-04-30T06:43:08.422ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
47682025-04-30T06:43:08.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2879511-f252-40f4-a587-b1550192746c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47692025-04-30T06:43:08.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
47702025-04-30T06:43:08.428ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e976b1-c7bf-4b43-ab9f-712b0208f2da", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47712025-04-30T06:43:08.428ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
47722025-04-30T06:43:08.429ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a00e2116-70d0-4b6f-90f3-bbbfaa934bd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47732025-04-30T06:43:08.429ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
47742025-04-30T06:43:08.432ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "491a438e-0d81-4d26-a0f3-a6530a7c38ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47752025-04-30T06:43:08.432ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
47762025-04-30T06:43:08.436ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb003d79-629e-48c1-b8ef-2c838ec179fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47772025-04-30T06:43:08.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
47782025-04-30T06:43:08.439ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea1feb5f-baf5-49f7-bfbe-96bce569046f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47792025-04-30T06:43:08.440ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
47802025-04-30T06:43:08.442ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "563d5910-082f-4dab-8348-8354522c081b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47812025-04-30T06:43:08.442ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
47822025-04-30T06:43:08.445ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8579c6a-f3c9-4197-be4a-4443df75f724", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47832025-04-30T06:43:08.445ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
47842025-04-30T06:43:08.447ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3a6f2d-a275-4736-bfad-8ca9755fd375", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47852025-04-30T06:43:08.448ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
47862025-04-30T06:43:08.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "186083d3-0068-44ea-9f30-195fbd6c6b8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47872025-04-30T06:43:08.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
47882025-04-30T06:43:08.450ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "044bbe0e-6144-426e-80e3-741b342df7a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47892025-04-30T06:43:08.451ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
47902025-04-30T06:43:08.455ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d08aa1a-ed8d-4168-89c7-90b7093f4871", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47912025-04-30T06:43:08.455ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
47922025-04-30T06:43:08.466ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47932025-04-30T06:43:08.474ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47942025-04-30T06:43:08.474ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47952025-04-30T06:43:08.474ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
47962025-04-30T06:43:08.478ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47972025-04-30T06:43:08.478ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe8430a-c6fe-4414-977d-929834dbd882", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
47982025-04-30T06:43:08.478ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
47992025-04-30T06:43:08.481ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d9ad948-d442-4f77-a180-9638ec1a5cd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48002025-04-30T06:43:08.481ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
48012025-04-30T06:43:08.482ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48022025-04-30T06:43:08.492ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48032025-04-30T06:43:08.492ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48042025-04-30T06:43:08.492ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
48052025-04-30T06:43:08.493ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48062025-04-30T06:43:08.496ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48072025-04-30T06:43:08.497ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
48082025-04-30T06:43:08.497ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48092025-04-30T06:43:08.498ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6f301d-67cf-45ac-bda5-27b0313f4b77", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48102025-04-30T06:43:08.498ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
48112025-04-30T06:43:08.501ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d6b1b4-0ea5-4329-860c-6b0f82bd7bc3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48122025-04-30T06:43:08.502ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
48132025-04-30T06:43:08.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7239c64-d742-4a7f-96c0-bba96ca01eac", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48142025-04-30T06:43:08.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
48152025-04-30T06:43:08.514ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a534954-a616-40d9-8ac0-ca4206608c4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48162025-04-30T06:43:08.514ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
48172025-04-30T06:43:08.516ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "580bc7b4-2d40-4e2c-845c-56900a924b8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48182025-04-30T06:43:08.516ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
48192025-04-30T06:43:08.517ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93f7fbfe-4d67-46f2-82ec-792a0fb46f23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48202025-04-30T06:43:08.518ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
48212025-04-30T06:43:08.520ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a4695a0-9b21-43ee-9bd3-2911d2b65bf3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48222025-04-30T06:43:08.520ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
48232025-04-30T06:43:08.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29fbcff6-4985-47af-bb8f-c591bc5cce4f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48242025-04-30T06:43:08.532ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
48252025-04-30T06:43:08.538ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a034220-7767-42ed-8a92-84ad3092d41d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48262025-04-30T06:43:08.538ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
48272025-04-30T06:43:08.544ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
48282025-04-30T06:43:08.550ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2515f953-8232-436c-ac63-8064e7750de5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48292025-04-30T06:43:08.550ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
48302025-04-30T06:43:08.556ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70b104d8-17cd-4f31-8124-731169eaf811", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48312025-04-30T06:43:08.556ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
48322025-04-30T06:43:08.559ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b9b7f5-d97f-44d3-aaab-a532094aabf0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48332025-04-30T06:43:08.559ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
48342025-04-30T06:43:08.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dfc96ab-1039-4ce1-abc7-c999e5d1466c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48352025-04-30T06:43:08.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
48362025-04-30T06:43:08.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f6bd3b-9907-4cf8-a664-ef0a0f06d3f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48372025-04-30T06:43:08.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
48382025-04-30T06:43:08.566ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf72b7c0-a726-4577-b710-dce854173ad5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48392025-04-30T06:43:08.566ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
48402025-04-30T06:43:08.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc14d249-8b0b-4574-8241-4c013fafce16", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48412025-04-30T06:43:08.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
48422025-04-30T06:43:08.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8472cceb-5fc6-41fb-be42-229ec9c31539", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48432025-04-30T06:43:08.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
48442025-04-30T06:43:08.572ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4480580c-277c-4c42-82c3-b1bd1331ebd2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:07 GMT"} })
48452025-04-30T06:43:08.572ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48462025-04-30T06:43:08.572ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48472025-04-30T06:43:08.572ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48482025-04-30T06:43:08.572ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48492025-04-30T06:43:08.573ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48502025-04-30T06:43:08.576ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48512025-04-30T06:43:08.580ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48522025-04-30T06:43:08.580ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48532025-04-30T06:43:08.580ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48542025-04-30T06:43:08.580ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48552025-04-30T06:43:08.581ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48562025-04-30T06:43:08.581ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48572025-04-30T06:43:08.599ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48582025-04-30T06:43:08.599ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48592025-04-30T06:43:08.599ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48602025-04-30T06:43:08.599ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
48612025-04-30T06:43:08.600ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48622025-04-30T06:43:08.608ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48632025-04-30T06:43:08.621ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48642025-04-30T06:43:08.621ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48652025-04-30T06:43:08.621ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48662025-04-30T06:43:08.621ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48672025-04-30T06:43:08.625ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48682025-04-30T06:43:08.631ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48692025-04-30T06:43:08.631ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48702025-04-30T06:43:08.631ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
48712025-04-30T06:43:08.632ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48722025-04-30T06:43:08.688ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
48732025-04-30T06:43:08.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
48742025-04-30T06:43:08.689ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:50233 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
48752025-04-30T06:43:08.690ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:50233 req_id = 16db08d1-7320-4526-977b-84437fc01e24 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48762025-04-30T06:43:08.692ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1821 local_addr = [::1]:45714 method = GET remote_addr = [::1]:50233 req_id = 16db08d1-7320-4526-977b-84437fc01e24 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48772025-04-30T06:43:08.692ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48782025-04-30T06:43:08.692ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
48792025-04-30T06:43:08.693ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48802025-04-30T06:43:08.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "16db08d1-7320-4526-977b-84437fc01e24", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
48812025-04-30T06:43:08.696ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48822025-04-30T06:43:08.696ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48832025-04-30T06:43:08.696ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
48842025-04-30T06:43:08.699ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48852025-04-30T06:43:08.700ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48862025-04-30T06:43:08.700ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48872025-04-30T06:43:08.700ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48882025-04-30T06:43:08.703ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48892025-04-30T06:43:08.704ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48902025-04-30T06:43:08.705ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48912025-04-30T06:43:08.708ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48922025-04-30T06:43:08.710ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48932025-04-30T06:43:08.710ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48942025-04-30T06:43:08.710ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48952025-04-30T06:43:08.710ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48962025-04-30T06:43:08.716ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
48972025-04-30T06:43:08.717ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48982025-04-30T06:43:08.717ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48992025-04-30T06:43:08.718ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49002025-04-30T06:43:08.777ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49012025-04-30T06:43:08.778ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49022025-04-30T06:43:08.778ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 512.100036ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49032025-04-30T06:43:08.779ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49042025-04-30T06:43:08.779ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49052025-04-30T06:43:08.779ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49062025-04-30T06:43:08.779ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49072025-04-30T06:43:08.779ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49082025-04-30T06:43:08.779ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49092025-04-30T06:43:08.781ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
49102025-04-30T06:43:08.800ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49112025-04-30T06:43:08.816ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49122025-04-30T06:43:08.816ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49132025-04-30T06:43:08.816ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49142025-04-30T06:43:08.817ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49152025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
49162025-04-30T06:43:08.824ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49172025-04-30T06:43:08.824ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 59 9c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Y............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55743 req_id = 2b5ad9d8-ca6d-487a-8118-85272ff9fa1a
49182025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22940,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55743 req_id = 2b5ad9d8-ca6d-487a-8118-85272ff9fa1a
49192025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
49202025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49212025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55743 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2b5ad9d8-ca6d-487a-8118-85272ff9fa1a
49222025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4253, tv_nsec: 621911924 } })
49232025-04-30T06:43:08.824ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
49242025-04-30T06:43:08.824ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 80 1e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64714 req_id = bde8bab6-f7e3-484a-8ca3-5237285645af
49252025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32798,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64714 req_id = bde8bab6-f7e3-484a-8ca3-5237285645af
49262025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49272025-04-30T06:43:08.824ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64714 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bde8bab6-f7e3-484a-8ca3-5237285645af
49282025-04-30T06:43:08.886ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49292025-04-30T06:43:08.887ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49302025-04-30T06:43:08.887ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49312025-04-30T06:43:08.887ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49322025-04-30T06:43:08.887ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
49332025-04-30T06:43:08.887ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49342025-04-30T06:43:08.888ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5e 8f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ^............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56824 req_id = c5f959f2-3b32-486b-aae7-bd5c7997a363
49352025-04-30T06:43:08.888ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24207,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56824 req_id = c5f959f2-3b32-486b-aae7-bd5c7997a363
49362025-04-30T06:43:08.888ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
49372025-04-30T06:43:08.889ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24207, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56824 req_id = c5f959f2-3b32-486b-aae7-bd5c7997a363
49382025-04-30T06:43:08.889ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
49392025-04-30T06:43:08.889ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
49402025-04-30T06:43:08.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
49412025-04-30T06:43:08.890ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49422025-04-30T06:43:08.902ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49432025-04-30T06:43:08.902ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49442025-04-30T06:43:08.902ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
49452025-04-30T06:43:08.903ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49462025-04-30T06:43:08.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f3a13ec-5083-46cb-aaf9-4c48b308bcbd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49472025-04-30T06:43:08.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
49482025-04-30T06:43:08.906ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49492025-04-30T06:43:08.906ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
49502025-04-30T06:43:08.907ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49512025-04-30T06:43:08.908ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2670fbac-c3ec-4895-b4a2-2c5bd0a1b7b3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49522025-04-30T06:43:08.909ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
49532025-04-30T06:43:08.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd5f150-4455-4536-8b4a-b927657de076", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49542025-04-30T06:43:08.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
49552025-04-30T06:43:08.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab29122-b6bc-4403-867b-2fdb8e358e0e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49562025-04-30T06:43:08.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
49572025-04-30T06:43:08.918ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceaec089-0b56-49d7-a4f5-e07b0d2dda20", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49582025-04-30T06:43:08.918ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
49592025-04-30T06:43:08.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0d122b-1cf6-4eeb-9039-6859a96dd8ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49602025-04-30T06:43:08.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
49612025-04-30T06:43:08.922ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c00a21-081b-489d-b572-ac3ca58f9e83", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49622025-04-30T06:43:08.923ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
49632025-04-30T06:43:08.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d54990ab-2426-420a-aaab-5c3ae97cd1d1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49642025-04-30T06:43:08.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
49652025-04-30T06:43:08.926ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1a865c3-9b66-40d5-a732-731bab94c189", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49662025-04-30T06:43:08.926ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
49672025-04-30T06:43:08.927ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a85aacc-303f-48be-8f7e-82cc71a541c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49682025-04-30T06:43:08.927ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
49692025-04-30T06:43:08.928ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7dd69c4-10d6-4b04-b157-83bfbe4d9edd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49702025-04-30T06:43:08.928ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
49712025-04-30T06:43:08.929ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4485aab-c6d2-4399-9fce-b41eabca60c6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49722025-04-30T06:43:08.930ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
49732025-04-30T06:43:08.939ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
49742025-04-30T06:43:08.939ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24e0f32a-a42f-4505-b128-7d1217cf23c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49752025-04-30T06:43:08.939ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
49762025-04-30T06:43:08.942ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d131cb2b-710b-4861-a52f-3ebfc154e5cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49772025-04-30T06:43:08.942ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
49782025-04-30T06:43:08.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "754eb94f-4ce4-485d-a268-e563f996590f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49792025-04-30T06:43:08.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
49802025-04-30T06:43:08.946ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec97bea8-7427-4b81-8064-da4e0bbdb9e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49812025-04-30T06:43:08.946ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
49822025-04-30T06:43:08.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b613e6-8bd4-455f-ac87-7686304476ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49832025-04-30T06:43:08.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
49842025-04-30T06:43:08.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3c0f28e-f8a6-49cf-8105-d8aa10b75a81", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49852025-04-30T06:43:08.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
49862025-04-30T06:43:08.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad33e21-846d-49bf-ac3a-4483d4cdf0d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49872025-04-30T06:43:08.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
49882025-04-30T06:43:08.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "887d7b0d-5355-4774-8957-e4ec4c6cdd9e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49892025-04-30T06:43:08.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
49902025-04-30T06:43:08.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c0dcdff-fd80-4e97-bcd5-c0ace5a5a72a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49912025-04-30T06:43:08.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
49922025-04-30T06:43:08.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84066acf-782f-4a86-8ccf-197bfa61274c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49932025-04-30T06:43:08.958ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
49942025-04-30T06:43:08.959ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46827fdd-414e-4ee1-84c9-969b3b797e3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49952025-04-30T06:43:08.959ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
49962025-04-30T06:43:08.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5776325a-fb5b-45bf-a212-7defc4bd3a16", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49972025-04-30T06:43:08.962ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
49982025-04-30T06:43:08.963ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5407d5f-f67f-45e3-a742-5113204e0287", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
49992025-04-30T06:43:08.963ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
50002025-04-30T06:43:08.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdfcedf4-d1b6-4341-a45f-8553cbd000cd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50012025-04-30T06:43:08.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
50022025-04-30T06:43:08.969ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fce34f88-6d7b-4877-b842-4dab0ae3fa19", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50032025-04-30T06:43:08.969ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
50042025-04-30T06:43:08.971ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f60da4-108c-4e5a-8c8c-d33b4944b623", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50052025-04-30T06:43:08.971ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
50062025-04-30T06:43:08.981ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50072025-04-30T06:43:08.981ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50082025-04-30T06:43:08.983ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
50092025-04-30T06:43:08.983ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50102025-04-30T06:43:08.988ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c45a943-76fc-4b75-991c-74da18818f41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50112025-04-30T06:43:08.988ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
50122025-04-30T06:43:08.988ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c90271-6364-4a03-a1e7-ed09f2a5bfa0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50132025-04-30T06:43:08.988ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
50142025-04-30T06:43:08.989ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
50152025-04-30T06:43:08.989ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50162025-04-30T06:43:08.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15dc3a8e-3ee1-4df4-bad1-02d16e95c444", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50172025-04-30T06:43:08.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
50182025-04-30T06:43:08.991ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50192025-04-30T06:43:08.992ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50202025-04-30T06:43:08.992ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50212025-04-30T06:43:08.992ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
50222025-04-30T06:43:08.992ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50232025-04-30T06:43:08.992ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
50242025-04-30T06:43:08.992ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50252025-04-30T06:43:08.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3a270f-f01b-4d8a-869d-ff7c2093fb4d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50262025-04-30T06:43:08.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
50272025-04-30T06:43:08.993ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50282025-04-30T06:43:09.014ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50292025-04-30T06:43:09.014ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50302025-04-30T06:43:09.014ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
50312025-04-30T06:43:09.015ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50322025-04-30T06:43:09.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed46ad71-4d58-417f-8f84-cc0640638034", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50332025-04-30T06:43:09.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
50342025-04-30T06:43:09.018ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50352025-04-30T06:43:09.018ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
50362025-04-30T06:43:09.019ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50372025-04-30T06:43:09.019ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "294e0efa-ed1b-4a89-9cfc-7f5760156a48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50382025-04-30T06:43:09.019ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
50392025-04-30T06:43:09.021ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b757f630-3df4-46dd-9956-9fa0762cd8f4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50402025-04-30T06:43:09.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
50412025-04-30T06:43:09.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8f53c0-b145-4631-86cf-c63245d78d8b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50422025-04-30T06:43:09.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
50432025-04-30T06:43:09.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8276a1e5-d114-41fc-91af-2df67b987cbd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50442025-04-30T06:43:09.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
50452025-04-30T06:43:09.032ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75da9ea7-ffd4-4f95-a594-466dc90c84ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50462025-04-30T06:43:09.032ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
50472025-04-30T06:43:09.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b4ff30-3845-4593-92b1-9739574029d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50482025-04-30T06:43:09.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
50492025-04-30T06:43:09.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92af0ada-684c-4536-b5d2-c9871d0653cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50502025-04-30T06:43:09.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
50512025-04-30T06:43:09.037ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dda872c-4177-40ea-bf28-51e2007cf00b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50522025-04-30T06:43:09.038ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
50532025-04-30T06:43:09.039ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed3f0dbb-59e2-497f-9efe-9280c8bf02b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50542025-04-30T06:43:09.039ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
50552025-04-30T06:43:09.040ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07c12ace-46eb-4b4d-a51c-882994765e07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50562025-04-30T06:43:09.040ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
50572025-04-30T06:43:09.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03bd5516-26b1-4f94-a9e9-c5d842a7e5e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50582025-04-30T06:43:09.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
50592025-04-30T06:43:09.043ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39c928e1-2f81-40c2-82ea-c0a15947d734", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50602025-04-30T06:43:09.043ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50612025-04-30T06:43:09.043ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50622025-04-30T06:43:09.043ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50632025-04-30T06:43:09.043ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50642025-04-30T06:43:09.044ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50652025-04-30T06:43:09.047ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
50662025-04-30T06:43:09.048ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50672025-04-30T06:43:09.053ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50682025-04-30T06:43:09.053ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50692025-04-30T06:43:09.054ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50702025-04-30T06:43:09.054ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50712025-04-30T06:43:09.054ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50722025-04-30T06:43:09.054ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50732025-04-30T06:43:09.057ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50742025-04-30T06:43:09.062ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50752025-04-30T06:43:09.062ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50762025-04-30T06:43:09.062ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50772025-04-30T06:43:09.063ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50782025-04-30T06:43:09.152ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
50792025-04-30T06:43:09.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
50802025-04-30T06:43:09.153ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50812025-04-30T06:43:09.153ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50822025-04-30T06:43:09.153ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50832025-04-30T06:43:09.153ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
50842025-04-30T06:43:09.154ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50852025-04-30T06:43:09.154ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:50840 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
50862025-04-30T06:43:09.154ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50872025-04-30T06:43:09.155ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:50840 req_id = 5871127d-da39-4914-936d-09c5ca4679f2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50882025-04-30T06:43:09.155ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 565 local_addr = [::1]:45714 method = GET remote_addr = [::1]:50840 req_id = 5871127d-da39-4914-936d-09c5ca4679f2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50892025-04-30T06:43:09.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5871127d-da39-4914-936d-09c5ca4679f2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
50902025-04-30T06:43:09.158ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50912025-04-30T06:43:09.159ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50922025-04-30T06:43:09.159ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
50932025-04-30T06:43:09.161ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
50942025-04-30T06:43:09.161ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50952025-04-30T06:43:09.161ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50962025-04-30T06:43:09.161ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50972025-04-30T06:43:09.161ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50982025-04-30T06:43:09.162ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50992025-04-30T06:43:09.189ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51002025-04-30T06:43:09.189ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51012025-04-30T06:43:09.189ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
51022025-04-30T06:43:09.189ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51032025-04-30T06:43:09.192ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51042025-04-30T06:43:09.196ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51052025-04-30T06:43:09.196ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51062025-04-30T06:43:09.196ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51072025-04-30T06:43:09.197ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51082025-04-30T06:43:09.198ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51092025-04-30T06:43:09.198ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
51102025-04-30T06:43:09.199ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51112025-04-30T06:43:09.278ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51122025-04-30T06:43:09.278ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51132025-04-30T06:43:09.278ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 500.198802ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51142025-04-30T06:43:09.279ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51152025-04-30T06:43:09.279ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51162025-04-30T06:43:09.279ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51172025-04-30T06:43:09.279ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51182025-04-30T06:43:09.279ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51192025-04-30T06:43:09.280ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51202025-04-30T06:43:09.288ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51212025-04-30T06:43:09.294ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51222025-04-30T06:43:09.294ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51232025-04-30T06:43:09.294ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51242025-04-30T06:43:09.295ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51252025-04-30T06:43:09.308ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
51262025-04-30T06:43:09.314ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
51272025-04-30T06:43:09.314ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51282025-04-30T06:43:09.318ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c5 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64969 req_id = dd943499-3296-493b-b475-ee528b1ab7af
51292025-04-30T06:43:09.318ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50506,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64969 req_id = dd943499-3296-493b-b475-ee528b1ab7af
51302025-04-30T06:43:09.318ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
51312025-04-30T06:43:09.318ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51322025-04-30T06:43:09.318ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64969 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dd943499-3296-493b-b475-ee528b1ab7af
51332025-04-30T06:43:09.320ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4254, tv_nsec: 118927996 } })
51342025-04-30T06:43:09.320ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
51352025-04-30T06:43:09.323ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2f e1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e /...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53795 req_id = 56b063bc-c7e8-4909-9cc3-b68b82b434e4
51362025-04-30T06:43:09.323ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12257,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53795 req_id = 56b063bc-c7e8-4909-9cc3-b68b82b434e4
51372025-04-30T06:43:09.323ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51382025-04-30T06:43:09.323ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53795 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 56b063bc-c7e8-4909-9cc3-b68b82b434e4
51392025-04-30T06:43:09.377ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51402025-04-30T06:43:09.384ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51412025-04-30T06:43:09.385ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51422025-04-30T06:43:09.385ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51432025-04-30T06:43:09.385ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
51442025-04-30T06:43:09.385ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51452025-04-30T06:43:09.391ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a 03 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50730 req_id = d3bc3aa7-a55f-4af4-8b9e-136178666c0b
51462025-04-30T06:43:09.391ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50730 req_id = d3bc3aa7-a55f-4af4-8b9e-136178666c0b
51472025-04-30T06:43:09.391ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
51482025-04-30T06:43:09.391ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39427, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50730 req_id = d3bc3aa7-a55f-4af4-8b9e-136178666c0b
51492025-04-30T06:43:09.392ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
51502025-04-30T06:43:09.392ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
51512025-04-30T06:43:09.392ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
51522025-04-30T06:43:09.404ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51532025-04-30T06:43:09.411ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51542025-04-30T06:43:09.411ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51552025-04-30T06:43:09.411ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
51562025-04-30T06:43:09.411ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51572025-04-30T06:43:09.414ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5e573d7-35f4-4209-9c5a-417f6869139b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51582025-04-30T06:43:09.414ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
51592025-04-30T06:43:09.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51602025-04-30T06:43:09.415ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
51612025-04-30T06:43:09.416ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51622025-04-30T06:43:09.418ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2208db35-499d-43b2-b646-d787d8596b33", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51632025-04-30T06:43:09.418ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
51642025-04-30T06:43:09.423ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff891fd-5925-4d00-86bb-7d60615cc333", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51652025-04-30T06:43:09.423ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
51662025-04-30T06:43:09.426ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "780f322a-6b78-4c95-b535-bafd2bc11edf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51672025-04-30T06:43:09.426ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
51682025-04-30T06:43:09.431ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b878c0ec-6ef6-410e-85fa-b748668d206d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51692025-04-30T06:43:09.431ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
51702025-04-30T06:43:09.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd06af9e-6da2-4dd9-9d8b-7d5c1ba5e40a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51712025-04-30T06:43:09.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
51722025-04-30T06:43:09.459ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d02953-85f4-46c1-8092-8ba6d7c69d34", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51732025-04-30T06:43:09.460ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
51742025-04-30T06:43:09.463ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
51752025-04-30T06:43:09.466ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ab1c947-5dcd-4959-9ea9-1280d7fd06f3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51762025-04-30T06:43:09.466ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
51772025-04-30T06:43:09.468ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "303a8f73-b073-48e1-8fa5-48129a5b4eca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51782025-04-30T06:43:09.468ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
51792025-04-30T06:43:09.471ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc548afd-18fc-4872-8463-bd6b51543f17", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51802025-04-30T06:43:09.471ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
51812025-04-30T06:43:09.473ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6f0c552-d6ef-40f5-b3a6-a382820967a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51822025-04-30T06:43:09.473ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
51832025-04-30T06:43:09.476ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34355619-de21-4ede-bb2e-d380577656e4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51842025-04-30T06:43:09.476ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
51852025-04-30T06:43:09.479ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2797aa59-2584-49fd-bea7-5c76a307b77c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51862025-04-30T06:43:09.479ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
51872025-04-30T06:43:09.482ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b7dfc3-debd-4f16-8da8-d1efd7ca89e3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51882025-04-30T06:43:09.482ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
51892025-04-30T06:43:09.484ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efe0b76d-2f69-4310-a887-1e0d1c4b9dca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51902025-04-30T06:43:09.484ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
51912025-04-30T06:43:09.487ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c762394-d30e-4f08-a7fd-740bc783ef24", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51922025-04-30T06:43:09.487ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
51932025-04-30T06:43:09.489ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36fd9c9-c4e2-4730-bdb3-0f03b638082d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51942025-04-30T06:43:09.489ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
51952025-04-30T06:43:09.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0b7388d-b193-422b-a97a-4646c4006e08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51962025-04-30T06:43:09.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
51972025-04-30T06:43:09.495ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1997cf-d422-42ae-aedb-17351ef2df38", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
51982025-04-30T06:43:09.495ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
51992025-04-30T06:43:09.497ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c021c4b8-1ede-4a0f-b3ad-3f50aae3cf5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52002025-04-30T06:43:09.497ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
52012025-04-30T06:43:09.499ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54c75e6e-5c9f-4d36-ad0b-e137244c6154", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52022025-04-30T06:43:09.499ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
52032025-04-30T06:43:09.501ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "188d5ad0-90c4-488e-bf46-3fc195c293e3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52042025-04-30T06:43:09.501ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
52052025-04-30T06:43:09.502ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f48a76e-a18e-4dd4-b1a7-ee250eb00b31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52062025-04-30T06:43:09.502ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
52072025-04-30T06:43:09.530ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f658a33-8457-47b6-966f-921aa1ec3bd9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52082025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
52092025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8c7730d-bc5c-4b72-bd74-be65ce8c57f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52102025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
52112025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df2f0ff8-6e97-4ce0-bb6c-fda80b360522", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52122025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
52132025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8d4e580-3999-4b93-9fe9-ee6323472b39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52142025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
52152025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9c0cc95-b8d9-4f20-8f32-f021bf86f8ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52162025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
52172025-04-30T06:43:09.531ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52182025-04-30T06:43:09.531ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52192025-04-30T06:43:09.531ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52202025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
52212025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52222025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcfcd642-91ee-4f3d-a34e-62842126bcbe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52232025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
52242025-04-30T06:43:09.531ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52252025-04-30T06:43:09.548ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52262025-04-30T06:43:09.548ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52272025-04-30T06:43:09.548ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
52282025-04-30T06:43:09.549ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52292025-04-30T06:43:09.554ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52302025-04-30T06:43:09.554ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
52312025-04-30T06:43:09.554ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52322025-04-30T06:43:09.557ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93bf2375-01dc-4e66-b839-bc02690032fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52332025-04-30T06:43:09.557ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
52342025-04-30T06:43:09.560ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fa7d85-d7e8-414e-a497-2fa06ecd1fdd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52352025-04-30T06:43:09.560ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
52362025-04-30T06:43:09.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb8dbb7-d1ff-4df4-acd3-930cb1c8aa7b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52372025-04-30T06:43:09.566ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
52382025-04-30T06:43:09.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a134d7-4568-4adc-bb3a-654a243a4d2e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52392025-04-30T06:43:09.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
52402025-04-30T06:43:09.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "557e2ece-4bd4-4421-8526-2975d108f47b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52412025-04-30T06:43:09.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
52422025-04-30T06:43:09.573ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd322f5-7e1c-422c-b36d-74e1bbb4ab9d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52432025-04-30T06:43:09.573ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
52442025-04-30T06:43:09.579ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b736e5d-faa5-4948-96b6-e837b845767f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52452025-04-30T06:43:09.579ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
52462025-04-30T06:43:09.581ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c759fe7-6d91-47bf-bbfc-ae699d3ac9d4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52472025-04-30T06:43:09.582ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
52482025-04-30T06:43:09.584ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0d973bc-6c2f-4b90-bc82-e58af359c0a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52492025-04-30T06:43:09.584ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
52502025-04-30T06:43:09.589ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
52512025-04-30T06:43:09.592ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9289e106-eebd-4b1d-bf77-adb576298f27", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52522025-04-30T06:43:09.592ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
52532025-04-30T06:43:09.594ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60198b99-eb7f-47b2-bee5-f31095417f0a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52542025-04-30T06:43:09.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
52552025-04-30T06:43:09.598ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e88b7e-1099-4db7-a91c-a36d3a394243", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52562025-04-30T06:43:09.598ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
52572025-04-30T06:43:09.600ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "953f4621-34f0-48dc-ba32-d293d37f87df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52582025-04-30T06:43:09.600ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
52592025-04-30T06:43:09.601ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57925a1-2d8c-404e-b8f7-e3cfa1261961", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52602025-04-30T06:43:09.601ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
52612025-04-30T06:43:09.603ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16f0fe41-b321-4ea3-b412-4dafec3b775b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52622025-04-30T06:43:09.603ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
52632025-04-30T06:43:09.604ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1e4e7f-b422-40f5-b035-4f0d8d3d0649", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:08 GMT"} })
52642025-04-30T06:43:09.604ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52652025-04-30T06:43:09.604ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52662025-04-30T06:43:09.605ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52672025-04-30T06:43:09.605ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52682025-04-30T06:43:09.606ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52692025-04-30T06:43:09.621ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52702025-04-30T06:43:09.626ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52712025-04-30T06:43:09.626ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52722025-04-30T06:43:09.626ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52732025-04-30T06:43:09.626ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52742025-04-30T06:43:09.626ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52752025-04-30T06:43:09.627ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52762025-04-30T06:43:09.630ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52772025-04-30T06:43:09.636ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52782025-04-30T06:43:09.636ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52792025-04-30T06:43:09.636ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52802025-04-30T06:43:09.636ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52812025-04-30T06:43:09.637ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
52822025-04-30T06:43:09.638ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52832025-04-30T06:43:09.638ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52842025-04-30T06:43:09.639ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52852025-04-30T06:43:09.639ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52862025-04-30T06:43:09.639ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52872025-04-30T06:43:09.709ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
52882025-04-30T06:43:09.709ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
52892025-04-30T06:43:09.710ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 72014 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52902025-04-30T06:43:09.711ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52912025-04-30T06:43:09.711ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52922025-04-30T06:43:09.711ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52932025-04-30T06:43:09.711ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
52942025-04-30T06:43:09.712ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52952025-04-30T06:43:09.712ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:59794 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
52962025-04-30T06:43:09.712ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52972025-04-30T06:43:09.713ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9e90cc7-2f4b-46b8-9fda-fa62b860f2c3", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
52982025-04-30T06:43:09.713ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
52992025-04-30T06:43:09.713ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53002025-04-30T06:43:09.714ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 82d1be37-3679-4237-b43b-c816c183641c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
53012025-04-30T06:43:09.714ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:59794 req_id = 32b56c9f-e3c8-4227-8b82-ea5c53dc2f5a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53022025-04-30T06:43:09.714ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 82d1be37-3679-4237-b43b-c816c183641c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
53032025-04-30T06:43:09.714ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 82d1be37-3679-4237-b43b-c816c183641c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
53042025-04-30T06:43:09.715ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 82d1be37-3679-4237-b43b-c816c183641c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
53052025-04-30T06:43:09.716ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 2063 local_addr = [::1]:45714 method = GET remote_addr = [::1]:59794 req_id = 32b56c9f-e3c8-4227-8b82-ea5c53dc2f5a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53062025-04-30T06:43:09.717ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53072025-04-30T06:43:09.718ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53082025-04-30T06:43:09.718ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53092025-04-30T06:43:09.718ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53102025-04-30T06:43:09.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "32b56c9f-e3c8-4227-8b82-ea5c53dc2f5a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
53112025-04-30T06:43:09.720ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53122025-04-30T06:43:09.720ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53132025-04-30T06:43:09.720ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
53142025-04-30T06:43:09.721ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53152025-04-30T06:43:09.721ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53162025-04-30T06:43:09.721ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53172025-04-30T06:43:09.721ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53182025-04-30T06:43:09.722ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53192025-04-30T06:43:09.722ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53202025-04-30T06:43:09.735ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53212025-04-30T06:43:09.735ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53222025-04-30T06:43:09.735ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
53232025-04-30T06:43:09.736ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53242025-04-30T06:43:09.737ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 23406 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 82d1be37-3679-4237-b43b-c816c183641c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
53252025-04-30T06:43:09.740ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53262025-04-30T06:43:09.746ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53272025-04-30T06:43:09.746ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53282025-04-30T06:43:09.746ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53292025-04-30T06:43:09.747ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53302025-04-30T06:43:09.750ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d1be37-3679-4237-b43b-c816c183641c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
53312025-04-30T06:43:09.750ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53322025-04-30T06:43:09.751ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
53332025-04-30T06:43:09.751ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
53342025-04-30T06:43:09.751ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53352025-04-30T06:43:09.751ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
53362025-04-30T06:43:09.752ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53372025-04-30T06:43:09.908ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
53382025-04-30T06:43:09.964ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53392025-04-30T06:43:09.964ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53402025-04-30T06:43:09.964ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53412025-04-30T06:43:09.965ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
53422025-04-30T06:43:09.969ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53432025-04-30T06:43:09.980ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53442025-04-30T06:43:09.981ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53452025-04-30T06:43:09.981ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 702.339716ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53462025-04-30T06:43:09.981ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53472025-04-30T06:43:09.981ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53482025-04-30T06:43:09.981ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53492025-04-30T06:43:09.981ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53502025-04-30T06:43:09.981ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53512025-04-30T06:43:09.982ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53522025-04-30T06:43:09.983ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53532025-04-30T06:43:09.983ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53542025-04-30T06:43:09.983ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
53552025-04-30T06:43:09.984ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
53562025-04-30T06:43:09.985ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572025-04-30T06:43:09.990ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53582025-04-30T06:43:09.993ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53592025-04-30T06:43:09.993ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
53602025-04-30T06:43:09.994ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53612025-04-30T06:43:09.995ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53622025-04-30T06:43:10.005ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53632025-04-30T06:43:10.016ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53642025-04-30T06:43:10.016ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53652025-04-30T06:43:10.053ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53662025-04-30T06:43:10.057ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53672025-04-30T06:43:10.057ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
53682025-04-30T06:43:10.057ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53692025-04-30T06:43:10.057ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
53702025-04-30T06:43:10.057ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
53712025-04-30T06:43:10.064ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
53722025-04-30T06:43:10.064ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53732025-04-30T06:43:10.068ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ea 5b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .[..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59980 req_id = bff1e6a1-7c34-41ad-b0e8-b0e183b3a3f7
53742025-04-30T06:43:10.068ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59980 req_id = bff1e6a1-7c34-41ad-b0e8-b0e183b3a3f7
53752025-04-30T06:43:10.068ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
53762025-04-30T06:43:10.081ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53772025-04-30T06:43:10.088ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59980 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bff1e6a1-7c34-41ad-b0e8-b0e183b3a3f7
53782025-04-30T06:43:10.089ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
53792025-04-30T06:43:10.089ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
53802025-04-30T06:43:10.089ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4254, tv_nsec: 888670027 } })
53812025-04-30T06:43:10.089ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
53822025-04-30T06:43:10.090ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
53832025-04-30T06:43:10.091ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55172 req_id = 8efe8b81-a1ae-4f6c-b575-d3df2f344688
53842025-04-30T06:43:10.091ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9241,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55172 req_id = 8efe8b81-a1ae-4f6c-b575-d3df2f344688
53852025-04-30T06:43:10.091ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53862025-04-30T06:43:10.094ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55172 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8efe8b81-a1ae-4f6c-b575-d3df2f344688
53872025-04-30T06:43:10.206ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53882025-04-30T06:43:10.207ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e3 ef 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54560 req_id = 4f93cd08-f7c1-4cd2-b7bc-b5d89c2b4e34
53892025-04-30T06:43:10.208ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58351,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54560 req_id = 4f93cd08-f7c1-4cd2-b7bc-b5d89c2b4e34
53902025-04-30T06:43:10.209ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
53912025-04-30T06:43:10.209ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58351, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54560 req_id = 4f93cd08-f7c1-4cd2-b7bc-b5d89c2b4e34
53922025-04-30T06:43:10.210ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
53932025-04-30T06:43:10.210ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
53942025-04-30T06:43:10.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
53952025-04-30T06:43:10.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4dc65e1-fd1c-45cf-af36-96056bc7e304", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
53962025-04-30T06:43:10.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
53972025-04-30T06:43:10.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b34d6768-0395-44fc-97cd-e6ee7d0f09eb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
53982025-04-30T06:43:10.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
53992025-04-30T06:43:10.234ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a4d0301-56b7-411e-83da-7b1a0e8205a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54002025-04-30T06:43:10.234ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
54012025-04-30T06:43:10.237ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "884a83cf-033b-4990-ab59-96b23c936806", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54022025-04-30T06:43:10.238ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
54032025-04-30T06:43:10.242ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
54042025-04-30T06:43:10.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74f503e9-1ef6-4b9d-8aff-85bb71284b23", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54052025-04-30T06:43:10.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
54062025-04-30T06:43:10.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d7e184b-0995-46f0-b5b8-85072b07dd00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54072025-04-30T06:43:10.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
54082025-04-30T06:43:10.285ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "541ac53a-c577-4645-af8c-b3bd6c78984f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54092025-04-30T06:43:10.285ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
54102025-04-30T06:43:10.297ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54112025-04-30T06:43:10.297ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54122025-04-30T06:43:10.298ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54132025-04-30T06:43:10.298ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
54142025-04-30T06:43:10.298ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54152025-04-30T06:43:10.306ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54162025-04-30T06:43:10.313ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54172025-04-30T06:43:10.313ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54182025-04-30T06:43:10.313ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
54192025-04-30T06:43:10.314ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54202025-04-30T06:43:10.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1022ca1b-024d-478a-b4a8-0bbd926877c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54212025-04-30T06:43:10.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
54222025-04-30T06:43:10.318ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54232025-04-30T06:43:10.318ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
54242025-04-30T06:43:10.319ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54252025-04-30T06:43:10.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1f6d687-d95d-4a27-b99a-a2f2713040c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54262025-04-30T06:43:10.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
54272025-04-30T06:43:10.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45fd7f4-6fd2-43e1-b4b6-724a7bcbe07b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54282025-04-30T06:43:10.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
54292025-04-30T06:43:10.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f1291ab-5583-4ba9-b4cb-63330c515e5e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54302025-04-30T06:43:10.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
54312025-04-30T06:43:10.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80cc99f6-20aa-40b8-b27d-dac7fde0ca74", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54322025-04-30T06:43:10.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
54332025-04-30T06:43:10.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "faee7106-23ed-4f07-b873-b1406a8f6016", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54342025-04-30T06:43:10.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
54352025-04-30T06:43:10.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78c2fe84-2481-4d61-88b9-e0c759e2bb6f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54362025-04-30T06:43:10.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
54372025-04-30T06:43:10.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb598823-ccc2-402f-a9e1-99986ac072df", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54382025-04-30T06:43:10.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
54392025-04-30T06:43:10.343ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5593c70a-c0ed-447f-a0d1-e45aad748d21", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54402025-04-30T06:43:10.343ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
54412025-04-30T06:43:10.348ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba00aef2-e4bd-46fa-9f97-e5c2ca4e81f8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54422025-04-30T06:43:10.349ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
54432025-04-30T06:43:10.354ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64e2608-720d-4165-8919-ec539b995a29", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54442025-04-30T06:43:10.354ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
54452025-04-30T06:43:10.360ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
54462025-04-30T06:43:10.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aa5783b-1304-4b27-8cba-13a6ff9d4a2c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54472025-04-30T06:43:10.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
54482025-04-30T06:43:10.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f580dc8-de39-495f-b5a1-4a0a4775d7d5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54492025-04-30T06:43:10.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
54502025-04-30T06:43:10.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "168c0b50-6029-4aef-9a8d-69d4f7ad54d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54512025-04-30T06:43:10.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
54522025-04-30T06:43:10.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fabcfbfc-e0a1-4f16-8273-4d68b662b831", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54532025-04-30T06:43:10.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
54542025-04-30T06:43:10.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4693df14-37f2-46d1-869e-a14661c15996", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54552025-04-30T06:43:10.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
54562025-04-30T06:43:10.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaf315f8-49c0-471a-a5ac-955661c174b8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54572025-04-30T06:43:10.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
54582025-04-30T06:43:10.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e08300d-3305-4697-82ed-fbc03fe694c4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54592025-04-30T06:43:10.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
54602025-04-30T06:43:10.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "131e8f48-7c50-45c9-aa7d-a0c6a30fc650", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54612025-04-30T06:43:10.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
54622025-04-30T06:43:10.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70f62cf1-0162-4d7f-bcac-d296d31cd5d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54632025-04-30T06:43:10.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
54642025-04-30T06:43:10.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37309ec0-eee2-4ad1-93de-3616bd24ba15", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54652025-04-30T06:43:10.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
54662025-04-30T06:43:10.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "354822ac-1df4-473c-a1ae-ce5c7a1fb4c5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54672025-04-30T06:43:10.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
54682025-04-30T06:43:10.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c1eebcf-41fe-4011-843a-f5cd925c74f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54692025-04-30T06:43:10.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
54702025-04-30T06:43:10.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26eca75c-2fba-43ba-8c00-e1e7b1e226ad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54712025-04-30T06:43:10.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
54722025-04-30T06:43:10.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed73b92e-2156-4980-b35e-06dde4b24389", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54732025-04-30T06:43:10.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
54742025-04-30T06:43:10.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11d36ac9-d7c1-4d73-b794-7394a2c2028e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54752025-04-30T06:43:10.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
54762025-04-30T06:43:10.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d60ef0-0ceb-4f0d-af63-c34ea44ef426", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54772025-04-30T06:43:10.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
54782025-04-30T06:43:10.392ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ef3f878-d2c9-4714-966f-6406d7ad7922", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54792025-04-30T06:43:10.393ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
54802025-04-30T06:43:10.394ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2ba712b-cfa5-4737-8c63-6186ddba9486", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54812025-04-30T06:43:10.396ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
54822025-04-30T06:43:10.405ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce6ae0c6-94c5-49e3-acc9-59f5de6755a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54832025-04-30T06:43:10.412ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
54842025-04-30T06:43:10.413ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54852025-04-30T06:43:10.413ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54862025-04-30T06:43:10.413ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54872025-04-30T06:43:10.413ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
54882025-04-30T06:43:10.413ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54892025-04-30T06:43:10.416ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "269d7e58-021d-483c-9089-6c0fa286f26e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54902025-04-30T06:43:10.416ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
54912025-04-30T06:43:10.417ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54922025-04-30T06:43:10.421ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54932025-04-30T06:43:10.421ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54942025-04-30T06:43:10.421ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
54952025-04-30T06:43:10.422ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54962025-04-30T06:43:10.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91578add-f4be-4e27-822c-3087ec1b9811", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
54972025-04-30T06:43:10.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
54982025-04-30T06:43:10.426ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54992025-04-30T06:43:10.426ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
55002025-04-30T06:43:10.426ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55012025-04-30T06:43:10.428ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12cfd1e2-3fbf-4712-b1b9-dfaed3761cf7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55022025-04-30T06:43:10.428ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
55032025-04-30T06:43:10.431ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e37255-2160-4545-890c-19e263f93c02", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55042025-04-30T06:43:10.432ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
55052025-04-30T06:43:10.433ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e801fae7-07cb-4f5a-9711-f4e9676e0917", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55062025-04-30T06:43:10.433ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
55072025-04-30T06:43:10.434ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c613fd6-4a51-4624-a689-33627ca16e83", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55082025-04-30T06:43:10.434ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
55092025-04-30T06:43:10.435ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c90571-6e34-4f1b-bb78-4363e0a8e87a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55102025-04-30T06:43:10.435ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
55112025-04-30T06:43:10.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5345ac8f-4fd2-444a-bca1-b145a97c654e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55122025-04-30T06:43:10.437ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55132025-04-30T06:43:10.437ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55142025-04-30T06:43:10.437ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55152025-04-30T06:43:10.437ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55162025-04-30T06:43:10.437ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55172025-04-30T06:43:10.458ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55182025-04-30T06:43:10.462ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55192025-04-30T06:43:10.462ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55202025-04-30T06:43:10.463ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55212025-04-30T06:43:10.463ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55222025-04-30T06:43:10.463ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55232025-04-30T06:43:10.464ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55242025-04-30T06:43:10.478ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55252025-04-30T06:43:10.481ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55262025-04-30T06:43:10.481ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55272025-04-30T06:43:10.481ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55282025-04-30T06:43:10.481ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55292025-04-30T06:43:10.569ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
55302025-04-30T06:43:10.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
55312025-04-30T06:43:10.574ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:35067 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55322025-04-30T06:43:10.575ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:35067 req_id = c15ab482-5b84-4ae6-a218-38bbe845b64d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55332025-04-30T06:43:10.576ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 980 local_addr = [::1]:45714 method = GET remote_addr = [::1]:35067 req_id = c15ab482-5b84-4ae6-a218-38bbe845b64d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55342025-04-30T06:43:10.576ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
55352025-04-30T06:43:10.577ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c15ab482-5b84-4ae6-a218-38bbe845b64d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:09 GMT"} })
55362025-04-30T06:43:10.590ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55372025-04-30T06:43:10.590ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55382025-04-30T06:43:10.590ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
55392025-04-30T06:43:10.620ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55402025-04-30T06:43:10.620ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55412025-04-30T06:43:10.621ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55422025-04-30T06:43:10.621ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55432025-04-30T06:43:10.622ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55442025-04-30T06:43:10.627ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55452025-04-30T06:43:10.637ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55462025-04-30T06:43:10.637ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55472025-04-30T06:43:10.638ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55482025-04-30T06:43:10.639ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55492025-04-30T06:43:10.639ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55502025-04-30T06:43:10.639ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55512025-04-30T06:43:10.639ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55522025-04-30T06:43:10.639ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
55532025-04-30T06:43:10.640ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55542025-04-30T06:43:10.652ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55552025-04-30T06:43:10.691ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55562025-04-30T06:43:10.691ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55572025-04-30T06:43:10.691ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
55582025-04-30T06:43:10.692ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55592025-04-30T06:43:10.695ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55602025-04-30T06:43:10.700ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
55612025-04-30T06:43:10.704ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55622025-04-30T06:43:10.707ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55632025-04-30T06:43:10.707ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55642025-04-30T06:43:10.712ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55652025-04-30T06:43:10.713ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55662025-04-30T06:43:10.713ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55672025-04-30T06:43:10.713ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55682025-04-30T06:43:10.818ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
55692025-04-30T06:43:10.843ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55702025-04-30T06:43:10.843ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55712025-04-30T06:43:10.843ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 861.995038ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55722025-04-30T06:43:10.843ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
55732025-04-30T06:43:10.843ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55742025-04-30T06:43:10.843ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55752025-04-30T06:43:10.843ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55762025-04-30T06:43:10.843ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55772025-04-30T06:43:10.844ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55782025-04-30T06:43:10.850ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55792025-04-30T06:43:10.857ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55802025-04-30T06:43:10.857ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55812025-04-30T06:43:10.857ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55822025-04-30T06:43:10.858ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55832025-04-30T06:43:10.862ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
55842025-04-30T06:43:10.862ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55852025-04-30T06:43:10.863ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 30 35 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 05..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50995 req_id = b06525ec-75d8-4d5a-ab6f-81f82284850d
55862025-04-30T06:43:10.863ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12341,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50995 req_id = b06525ec-75d8-4d5a-ab6f-81f82284850d
55872025-04-30T06:43:10.863ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
55882025-04-30T06:43:10.863ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55892025-04-30T06:43:10.863ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50995 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b06525ec-75d8-4d5a-ab6f-81f82284850d
55902025-04-30T06:43:10.863ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4255, tv_nsec: 662861031 } })
55912025-04-30T06:43:10.864ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
55922025-04-30T06:43:10.864ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2f 16 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e /...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64887 req_id = 8066b55b-12c4-4f2b-92a3-cb9ab8684205
55932025-04-30T06:43:10.864ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12054,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64887 req_id = 8066b55b-12c4-4f2b-92a3-cb9ab8684205
55942025-04-30T06:43:10.864ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55952025-04-30T06:43:10.864ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64887 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8066b55b-12c4-4f2b-92a3-cb9ab8684205
55962025-04-30T06:43:10.936ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55972025-04-30T06:43:10.936ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55982025-04-30T06:43:10.936ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55992025-04-30T06:43:10.936ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56002025-04-30T06:43:10.936ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56012025-04-30T06:43:10.937ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56022025-04-30T06:43:10.938ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5a c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Z............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59502 req_id = fa8e3df0-bf3c-46b7-8f55-f363735f0e88
56032025-04-30T06:43:10.938ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23232,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59502 req_id = fa8e3df0-bf3c-46b7-8f55-f363735f0e88
56042025-04-30T06:43:10.938ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
56052025-04-30T06:43:10.938ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23232, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59502 req_id = fa8e3df0-bf3c-46b7-8f55-f363735f0e88
56062025-04-30T06:43:10.938ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
56072025-04-30T06:43:10.938ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
56082025-04-30T06:43:10.938ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
56092025-04-30T06:43:10.941ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56102025-04-30T06:43:10.949ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56112025-04-30T06:43:10.949ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56122025-04-30T06:43:10.949ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56132025-04-30T06:43:10.950ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56142025-04-30T06:43:10.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5c9add-b8df-4f89-a95b-9ef4f71086bb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56152025-04-30T06:43:10.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
56162025-04-30T06:43:10.974ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56172025-04-30T06:43:10.974ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56182025-04-30T06:43:10.975ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56192025-04-30T06:43:10.975ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56202025-04-30T06:43:10.976ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56212025-04-30T06:43:10.976ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
56222025-04-30T06:43:10.980ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
56232025-04-30T06:43:10.982ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5030ee1f-ba16-456b-9042-30d9d55fc430", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56242025-04-30T06:43:10.983ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
56252025-04-30T06:43:11.017ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
56262025-04-30T06:43:11.017ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
56272025-04-30T06:43:11.027ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76b4930e-7b40-4b17-b226-a591a39e2a1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56282025-04-30T06:43:11.027ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
56292025-04-30T06:43:11.032ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
56302025-04-30T06:43:11.032ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56312025-04-30T06:43:11.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a792e21-19a0-4bbf-99b1-767a216c5dfe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56322025-04-30T06:43:11.042ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
56332025-04-30T06:43:11.048ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b31bfcd7-90b4-458f-86b9-efbcdeb67974", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56342025-04-30T06:43:11.048ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
56352025-04-30T06:43:11.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73c15dd1-8a60-4ea4-a1a5-b2febd64054f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56362025-04-30T06:43:11.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
56372025-04-30T06:43:11.064ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
56382025-04-30T06:43:11.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0417845a-ba16-42ed-9f5f-2b1916e3b77b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56392025-04-30T06:43:11.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
56402025-04-30T06:43:11.078ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
56412025-04-30T06:43:11.083ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8394a355-af84-45ca-878e-feedb99c50e5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56422025-04-30T06:43:11.083ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
56432025-04-30T06:43:11.084ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8dee916-5efb-477f-8dd9-17b9cbd29eda", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56442025-04-30T06:43:11.084ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
56452025-04-30T06:43:11.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e478785d-c12d-4fd8-a1a6-6e2d04f0cb9b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56462025-04-30T06:43:11.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
56472025-04-30T06:43:11.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d43cc07-3b0f-466e-98cb-dc1d3e30a939", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56482025-04-30T06:43:11.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
56492025-04-30T06:43:11.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6b544ff-6cfa-407e-b74b-295e013815f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56502025-04-30T06:43:11.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
56512025-04-30T06:43:11.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48361cbe-ff67-41e7-a618-0f5b2f8c3e3c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56522025-04-30T06:43:11.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
56532025-04-30T06:43:11.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7606d63-634c-4f35-8018-314ee89d4372", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56542025-04-30T06:43:11.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
56552025-04-30T06:43:11.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2998f42-0472-4601-a619-34f8d2e0977f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56562025-04-30T06:43:11.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
56572025-04-30T06:43:11.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b13bbf90-426f-424a-b180-dec86c92d53e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56582025-04-30T06:43:11.119ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
56592025-04-30T06:43:11.120ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56602025-04-30T06:43:11.121ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56612025-04-30T06:43:11.122ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56622025-04-30T06:43:11.123ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56632025-04-30T06:43:11.126ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56642025-04-30T06:43:11.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13b37ec5-1a35-491c-bcdc-2f19e6276da5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56652025-04-30T06:43:11.132ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
56662025-04-30T06:43:11.137ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56672025-04-30T06:43:11.149ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56682025-04-30T06:43:11.150ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56692025-04-30T06:43:11.150ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56702025-04-30T06:43:11.151ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56712025-04-30T06:43:11.154ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56722025-04-30T06:43:11.154ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
56732025-04-30T06:43:11.155ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56742025-04-30T06:43:11.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf704ec0-a6a3-46d0-876d-da58e2254d45", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56752025-04-30T06:43:11.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
56762025-04-30T06:43:11.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d64a76be-bf98-4de5-a43d-30488acc6948", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56772025-04-30T06:43:11.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
56782025-04-30T06:43:11.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "335d9707-5f7e-4789-9c44-03411655990a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56792025-04-30T06:43:11.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
56802025-04-30T06:43:11.170ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7e8f5c9-150a-45a4-801b-720da3afdbaf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56812025-04-30T06:43:11.170ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
56822025-04-30T06:43:11.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4636fcb0-7f18-4cd0-8d61-cf3dbd2e8c85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56832025-04-30T06:43:11.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
56842025-04-30T06:43:11.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46898aa1-b7ed-409c-8149-a47adbcffaeb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56852025-04-30T06:43:11.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
56862025-04-30T06:43:11.179ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4007c27-c4ae-479e-84a1-5bd0fe7cf60e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56872025-04-30T06:43:11.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
56882025-04-30T06:43:11.183ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
56892025-04-30T06:43:11.185ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "779d5942-6c1a-4a59-96b5-f004c63833a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56902025-04-30T06:43:11.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
56912025-04-30T06:43:11.188ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb3d4f0d-f941-4bd7-977d-a83f78a50432", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56922025-04-30T06:43:11.188ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
56932025-04-30T06:43:11.192ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5a8cbf-34af-465f-bfdf-1e6959ba9d18", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56942025-04-30T06:43:11.192ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
56952025-04-30T06:43:11.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1075d81-f514-4c1b-be46-ad5cb72be061", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56962025-04-30T06:43:11.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
56972025-04-30T06:43:11.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae44d96-b2b1-4b4a-a700-ff6a8cc80846", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
56982025-04-30T06:43:11.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
56992025-04-30T06:43:11.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c47dc59-c492-46f3-a128-94f190cee90c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57002025-04-30T06:43:11.207ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
57012025-04-30T06:43:11.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e6b8eb-432b-4a43-9663-aa1345a39ffc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57022025-04-30T06:43:11.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
57032025-04-30T06:43:11.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69c519db-de6c-4eed-8211-db9b501ec610", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57042025-04-30T06:43:11.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
57052025-04-30T06:43:11.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2574d647-8fe0-4318-88f8-f2f5f7951a78", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57062025-04-30T06:43:11.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
57072025-04-30T06:43:11.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3fa79e-d095-4bc9-abe7-b3e0e61eb610", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57082025-04-30T06:43:11.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
57092025-04-30T06:43:11.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ce40134-6d90-4e9b-b1aa-393ee6ab25da", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57102025-04-30T06:43:11.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
57112025-04-30T06:43:11.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ccbb07a-9acf-4d19-8278-83abfad09509", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57122025-04-30T06:43:11.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
57132025-04-30T06:43:11.236ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57142025-04-30T06:43:11.236ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57152025-04-30T06:43:11.236ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57162025-04-30T06:43:11.236ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57172025-04-30T06:43:11.237ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57182025-04-30T06:43:11.264ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57192025-04-30T06:43:11.271ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57202025-04-30T06:43:11.271ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57212025-04-30T06:43:11.271ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57222025-04-30T06:43:11.271ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57232025-04-30T06:43:11.272ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c9d7af-cf99-4385-af8d-83945e3871f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57242025-04-30T06:43:11.272ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
57252025-04-30T06:43:11.290ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57262025-04-30T06:43:11.290ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57272025-04-30T06:43:11.290ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57282025-04-30T06:43:11.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e74838-019a-4cec-bf29-7420d5de1e87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57292025-04-30T06:43:11.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
57302025-04-30T06:43:11.376ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
57312025-04-30T06:43:11.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "663aa76d-a162-4422-ba97-1f94800978dd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57322025-04-30T06:43:11.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
57332025-04-30T06:43:11.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafdeb0d-a556-405d-84ea-81ca4ee2282a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57342025-04-30T06:43:11.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
57352025-04-30T06:43:11.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e269d9a0-a5b2-4bde-88ec-c742732f0f64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57362025-04-30T06:43:11.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
57372025-04-30T06:43:11.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f24580a5-d3eb-4f47-98e2-0285e7b7c108", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57382025-04-30T06:43:11.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
57392025-04-30T06:43:11.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae8f8f5-7d2b-40cd-8c87-b40c225d07fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57402025-04-30T06:43:11.390ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
57412025-04-30T06:43:11.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50f77701-a8e6-44cc-b68a-f3cd86c2bcd8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57422025-04-30T06:43:11.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
57432025-04-30T06:43:11.392ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "974f63d6-d73c-4f1b-9f8f-32928a6cd3d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57442025-04-30T06:43:11.392ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57452025-04-30T06:43:11.392ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57462025-04-30T06:43:11.392ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57472025-04-30T06:43:11.392ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57482025-04-30T06:43:11.393ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57492025-04-30T06:43:11.396ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57502025-04-30T06:43:11.400ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57512025-04-30T06:43:11.400ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57522025-04-30T06:43:11.400ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57532025-04-30T06:43:11.400ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57542025-04-30T06:43:11.400ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57552025-04-30T06:43:11.401ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57562025-04-30T06:43:11.404ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57572025-04-30T06:43:11.411ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57582025-04-30T06:43:11.411ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57592025-04-30T06:43:11.411ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57602025-04-30T06:43:11.411ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57612025-04-30T06:43:11.462ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
57622025-04-30T06:43:11.462ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
57632025-04-30T06:43:11.463ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57642025-04-30T06:43:11.464ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57652025-04-30T06:43:11.465ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57662025-04-30T06:43:11.465ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57672025-04-30T06:43:11.468ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57682025-04-30T06:43:11.468ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:33552 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57692025-04-30T06:43:11.469ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:33552 req_id = 1bd47144-7596-4158-a59b-ffaf4cd68689 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57702025-04-30T06:43:11.470ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 575 local_addr = [::1]:45714 method = GET remote_addr = [::1]:33552 req_id = 1bd47144-7596-4158-a59b-ffaf4cd68689 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57712025-04-30T06:43:11.470ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bd47144-7596-4158-a59b-ffaf4cd68689", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
57722025-04-30T06:43:11.473ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57732025-04-30T06:43:11.473ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57742025-04-30T06:43:11.474ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
57752025-04-30T06:43:11.476ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57762025-04-30T06:43:11.476ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57772025-04-30T06:43:11.476ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57782025-04-30T06:43:11.476ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57792025-04-30T06:43:11.477ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57802025-04-30T06:43:11.477ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57812025-04-30T06:43:11.493ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57822025-04-30T06:43:11.493ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57832025-04-30T06:43:11.493ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57842025-04-30T06:43:11.494ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57852025-04-30T06:43:11.496ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57862025-04-30T06:43:11.504ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57872025-04-30T06:43:11.505ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57882025-04-30T06:43:11.505ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57892025-04-30T06:43:11.506ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57902025-04-30T06:43:11.506ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57912025-04-30T06:43:11.506ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
57922025-04-30T06:43:11.507ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57932025-04-30T06:43:11.537ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57942025-04-30T06:43:11.537ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57952025-04-30T06:43:11.537ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 693.617989ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57962025-04-30T06:43:11.537ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
57972025-04-30T06:43:11.537ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57982025-04-30T06:43:11.537ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57992025-04-30T06:43:11.537ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58002025-04-30T06:43:11.537ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58012025-04-30T06:43:11.538ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58022025-04-30T06:43:11.541ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
58032025-04-30T06:43:11.541ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58042025-04-30T06:43:11.544ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58052025-04-30T06:43:11.544ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58062025-04-30T06:43:11.544ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58072025-04-30T06:43:11.545ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58082025-04-30T06:43:11.548ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
58092025-04-30T06:43:11.548ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58102025-04-30T06:43:11.549ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 05 09 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65405 req_id = da470a59-ed59-4b38-8342-bbe3969ecce8
58112025-04-30T06:43:11.549ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1289,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65405 req_id = da470a59-ed59-4b38-8342-bbe3969ecce8
58122025-04-30T06:43:11.549ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
58132025-04-30T06:43:11.550ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58142025-04-30T06:43:11.550ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65405 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = da470a59-ed59-4b38-8342-bbe3969ecce8
58152025-04-30T06:43:11.550ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 349461057 } })
58162025-04-30T06:43:11.550ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
58172025-04-30T06:43:11.551ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 f3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56491 req_id = ea67c2eb-bebb-48b4-828e-f035255b5c16
58182025-04-30T06:43:11.551ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26099,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56491 req_id = ea67c2eb-bebb-48b4-828e-f035255b5c16
58192025-04-30T06:43:11.551ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58202025-04-30T06:43:11.551ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56491 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ea67c2eb-bebb-48b4-828e-f035255b5c16
58212025-04-30T06:43:11.606ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58222025-04-30T06:43:11.607ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58232025-04-30T06:43:11.607ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58242025-04-30T06:43:11.607ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58252025-04-30T06:43:11.607ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
58262025-04-30T06:43:11.608ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58272025-04-30T06:43:11.611ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d2 34 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .4..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56130 req_id = da9b4d4f-618d-433e-bb42-38517e8d7ef2
58282025-04-30T06:43:11.611ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53812,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56130 req_id = da9b4d4f-618d-433e-bb42-38517e8d7ef2
58292025-04-30T06:43:11.611ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
58302025-04-30T06:43:11.611ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53812, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56130 req_id = da9b4d4f-618d-433e-bb42-38517e8d7ef2
58312025-04-30T06:43:11.612ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
58322025-04-30T06:43:11.612ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
58332025-04-30T06:43:11.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
58342025-04-30T06:43:11.616ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58352025-04-30T06:43:11.623ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58362025-04-30T06:43:11.623ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58372025-04-30T06:43:11.623ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
58382025-04-30T06:43:11.623ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58392025-04-30T06:43:11.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "64856930-f700-4f14-b15e-51267dfc38c6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58402025-04-30T06:43:11.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
58412025-04-30T06:43:11.629ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58422025-04-30T06:43:11.629ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
58432025-04-30T06:43:11.630ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58442025-04-30T06:43:11.634ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d645c90-7aa0-4e40-b6bd-0046e742f68e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58452025-04-30T06:43:11.634ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
58462025-04-30T06:43:11.635ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
58472025-04-30T06:43:11.636ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58482025-04-30T06:43:11.637ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = f7169682-4391-489f-9280-123692cdeb4c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58492025-04-30T06:43:11.638ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = f7169682-4391-489f-9280-123692cdeb4c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58502025-04-30T06:43:11.638ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = f7169682-4391-489f-9280-123692cdeb4c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58512025-04-30T06:43:11.638ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = f7169682-4391-489f-9280-123692cdeb4c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58522025-04-30T06:43:11.643ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 6476 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = f7169682-4391-489f-9280-123692cdeb4c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58532025-04-30T06:43:11.643ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7169682-4391-489f-9280-123692cdeb4c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58542025-04-30T06:43:11.644ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58552025-04-30T06:43:11.644ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 57bec5a4-df5f-47d2-a85d-aab70e5a03a7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58562025-04-30T06:43:11.645ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 57bec5a4-df5f-47d2-a85d-aab70e5a03a7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58572025-04-30T06:43:11.645ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 57bec5a4-df5f-47d2-a85d-aab70e5a03a7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58582025-04-30T06:43:11.646ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 57bec5a4-df5f-47d2-a85d-aab70e5a03a7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58592025-04-30T06:43:11.652ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 7350 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 57bec5a4-df5f-47d2-a85d-aab70e5a03a7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
58602025-04-30T06:43:11.653ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "57bec5a4-df5f-47d2-a85d-aab70e5a03a7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58612025-04-30T06:43:11.653ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58622025-04-30T06:43:11.653ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
58632025-04-30T06:43:11.653ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
58642025-04-30T06:43:11.653ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd5f737-2c9d-42b9-9383-6c81c4fad218", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58652025-04-30T06:43:11.653ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
58662025-04-30T06:43:11.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb70499-8c98-4eab-9487-4fd025bcae95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58672025-04-30T06:43:11.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
58682025-04-30T06:43:11.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46131727-b81a-4471-9ca7-7d5f96723e6d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58692025-04-30T06:43:11.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
58702025-04-30T06:43:11.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41121ada-6cb3-443e-9852-28cb7fb33477", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58712025-04-30T06:43:11.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
58722025-04-30T06:43:11.677ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
58732025-04-30T06:43:11.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ffc7c8b-fe92-46d1-9285-ac85329d786c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58742025-04-30T06:43:11.681ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
58752025-04-30T06:43:11.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c40c9db-19f2-4121-86d8-5d294f338278", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58762025-04-30T06:43:11.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
58772025-04-30T06:43:11.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3ec067c-2c53-4c6c-b66d-7b633c43ff3a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58782025-04-30T06:43:11.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
58792025-04-30T06:43:11.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e82be2a-a17a-4294-9086-42bbe6ea7ffc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58802025-04-30T06:43:11.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
58812025-04-30T06:43:11.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4c44d20-d4d1-4083-a0bb-d761d1d27963", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58822025-04-30T06:43:11.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
58832025-04-30T06:43:11.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae32c76f-280e-46cb-8c9a-e66eb61bda50", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58842025-04-30T06:43:11.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
58852025-04-30T06:43:11.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97df2036-c198-4c71-906b-8b0b372e536c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58862025-04-30T06:43:11.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
58872025-04-30T06:43:11.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "302c57db-efaa-4631-a03d-da8c68af380b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58882025-04-30T06:43:11.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
58892025-04-30T06:43:11.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52b73145-2b45-430e-b329-2da5059f178e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58902025-04-30T06:43:11.700ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
58912025-04-30T06:43:11.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c2921db-18a5-4e04-80d0-7b1a80cae0b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
58922025-04-30T06:43:11.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
58932025-04-30T06:43:11.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58942025-04-30T06:43:11.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58952025-04-30T06:43:11.707ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58962025-04-30T06:43:11.709ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58972025-04-30T06:43:11.709ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58982025-04-30T06:43:11.709ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58992025-04-30T06:43:11.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3675244-b9b8-4acf-a76b-99f65b999153", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59002025-04-30T06:43:11.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
59012025-04-30T06:43:11.714ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1173e5b5-126a-457b-8333-48bb4836c6a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59022025-04-30T06:43:11.715ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
59032025-04-30T06:43:11.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f15471d1-a84c-419d-b29e-833ea9df04a1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59042025-04-30T06:43:11.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
59052025-04-30T06:43:11.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b40d016a-72fc-4710-a67a-3ae2a41a6959", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59062025-04-30T06:43:11.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
59072025-04-30T06:43:11.722ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "463e146a-257e-4c9d-8c4b-e291d1211da0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59082025-04-30T06:43:11.722ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
59092025-04-30T06:43:11.729ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59102025-04-30T06:43:11.729ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59112025-04-30T06:43:11.729ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59122025-04-30T06:43:11.729ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59132025-04-30T06:43:11.734ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59142025-04-30T06:43:11.737ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "952d6520-956d-43dd-bf51-dbc1f29743a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59152025-04-30T06:43:11.737ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
59162025-04-30T06:43:11.737ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59172025-04-30T06:43:11.750ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59182025-04-30T06:43:11.750ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59192025-04-30T06:43:11.750ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59202025-04-30T06:43:11.752ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59212025-04-30T06:43:11.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c78f914-1ff3-4259-85cb-f93318c97338", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59222025-04-30T06:43:11.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
59232025-04-30T06:43:11.758ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59242025-04-30T06:43:11.758ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59252025-04-30T06:43:11.758ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59262025-04-30T06:43:11.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0a5fb6b-83da-431a-b376-3123b066a93c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59272025-04-30T06:43:11.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
59282025-04-30T06:43:11.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17c95f8-fea9-4a20-8326-39463dadb5d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59292025-04-30T06:43:11.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
59302025-04-30T06:43:11.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef65f48e-a081-4f5d-b248-e174d7443328", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59312025-04-30T06:43:11.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
59322025-04-30T06:43:11.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2430414-42fc-4dc4-8abe-66aa46605f77", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59332025-04-30T06:43:11.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
59342025-04-30T06:43:11.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "486f38bd-3f1e-4b02-9e15-17e39670c31d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59352025-04-30T06:43:11.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
59362025-04-30T06:43:11.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c59278e-5918-4cff-9765-64f06f1bf71b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59372025-04-30T06:43:11.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
59382025-04-30T06:43:11.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2cf874e-1088-4c7b-a53e-fc4c4df492a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59392025-04-30T06:43:11.790ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
59402025-04-30T06:43:11.795ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0b48040-ac69-4f74-a59a-72bcd61cc627", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59412025-04-30T06:43:11.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
59422025-04-30T06:43:11.799ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
59432025-04-30T06:43:11.807ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e8ce73-4d77-4420-a6d0-79834bc306d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59442025-04-30T06:43:11.807ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
59452025-04-30T06:43:11.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f42034da-e2a3-4c32-8732-5ddfe2910917", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59462025-04-30T06:43:11.811ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
59472025-04-30T06:43:11.812ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c247fa20-7762-4dbf-9c65-ebe527a80f96", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59482025-04-30T06:43:11.813ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
59492025-04-30T06:43:11.815ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05096f8e-4477-42e4-b2d4-d9e8759d8647", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59502025-04-30T06:43:11.815ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
59512025-04-30T06:43:11.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4590d1d-58b2-4b5d-8aec-ccf8d98f4c85", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59522025-04-30T06:43:11.820ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
59532025-04-30T06:43:11.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96bfe1dd-742b-478a-aef1-31dc95cfc42a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59542025-04-30T06:43:11.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
59552025-04-30T06:43:11.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dc72708-1875-4c28-af09-e0abdcc77301", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59562025-04-30T06:43:11.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
59572025-04-30T06:43:11.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f15ff64-15d0-4c27-8cb9-ab5f0ce7232d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59582025-04-30T06:43:11.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
59592025-04-30T06:43:11.836ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3e4215-4c28-4fc5-ad60-d492da01dd2d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59602025-04-30T06:43:11.836ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
59612025-04-30T06:43:11.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db5fc6b7-0456-4d91-9f58-b5c37b7eae1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59622025-04-30T06:43:11.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
59632025-04-30T06:43:11.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1597eb20-1c3e-4109-8ab5-d8d185957496", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59642025-04-30T06:43:11.841ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
59652025-04-30T06:43:11.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21bc64ab-e3fa-4de4-9666-279e875c07ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59662025-04-30T06:43:11.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
59672025-04-30T06:43:11.845ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ec7b5b-5da4-4a89-9259-1851d6646d39", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59682025-04-30T06:43:11.845ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
59692025-04-30T06:43:11.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73be4959-c488-40fd-bd05-9113eab41acc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:10 GMT"} })
59702025-04-30T06:43:11.846ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59712025-04-30T06:43:11.847ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59722025-04-30T06:43:11.847ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59732025-04-30T06:43:11.847ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59742025-04-30T06:43:11.847ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59752025-04-30T06:43:11.850ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59762025-04-30T06:43:11.850ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59772025-04-30T06:43:11.850ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59782025-04-30T06:43:11.850ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59792025-04-30T06:43:11.851ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59802025-04-30T06:43:11.851ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59812025-04-30T06:43:11.854ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59822025-04-30T06:43:11.854ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59832025-04-30T06:43:11.854ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59842025-04-30T06:43:11.855ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59852025-04-30T06:43:11.855ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59862025-04-30T06:43:11.855ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59872025-04-30T06:43:11.857ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59882025-04-30T06:43:11.862ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59892025-04-30T06:43:11.862ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59902025-04-30T06:43:11.862ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59912025-04-30T06:43:11.863ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59922025-04-30T06:43:11.863ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59932025-04-30T06:43:11.871ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59942025-04-30T06:43:11.871ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59952025-04-30T06:43:11.871ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59962025-04-30T06:43:11.871ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59972025-04-30T06:43:11.875ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59982025-04-30T06:43:11.875ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
59992025-04-30T06:43:11.877ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60002025-04-30T06:43:11.948ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
60012025-04-30T06:43:11.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
60022025-04-30T06:43:11.949ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:63390 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60032025-04-30T06:43:11.950ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:63390 req_id = 1b6aae2a-65ee-4538-ada8-7060e07527f0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60042025-04-30T06:43:11.951ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 540 local_addr = [::1]:45714 method = GET remote_addr = [::1]:63390 req_id = 1b6aae2a-65ee-4538-ada8-7060e07527f0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60052025-04-30T06:43:11.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b6aae2a-65ee-4538-ada8-7060e07527f0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60062025-04-30T06:43:11.955ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60072025-04-30T06:43:11.955ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60082025-04-30T06:43:11.955ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
60092025-04-30T06:43:11.957ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60102025-04-30T06:43:11.957ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60112025-04-30T06:43:11.957ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60122025-04-30T06:43:11.957ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60132025-04-30T06:43:11.957ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60142025-04-30T06:43:11.983ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60152025-04-30T06:43:11.998ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60162025-04-30T06:43:11.998ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60172025-04-30T06:43:11.998ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60182025-04-30T06:43:11.999ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60192025-04-30T06:43:12.001ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60202025-04-30T06:43:12.001ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60212025-04-30T06:43:12.002ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
60222025-04-30T06:43:12.003ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
60232025-04-30T06:43:12.011ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
60242025-04-30T06:43:12.011ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
60252025-04-30T06:43:12.029ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
60262025-04-30T06:43:12.030ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60272025-04-30T06:43:12.088ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
60282025-04-30T06:43:12.094ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60292025-04-30T06:43:12.095ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60302025-04-30T06:43:12.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 557.816593ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60312025-04-30T06:43:12.095ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
60322025-04-30T06:43:12.095ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60332025-04-30T06:43:12.095ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60342025-04-30T06:43:12.095ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60352025-04-30T06:43:12.095ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60362025-04-30T06:43:12.096ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60372025-04-30T06:43:12.104ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
60382025-04-30T06:43:12.104ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60392025-04-30T06:43:12.111ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60402025-04-30T06:43:12.111ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60412025-04-30T06:43:12.111ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60422025-04-30T06:43:12.111ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60432025-04-30T06:43:12.115ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
60442025-04-30T06:43:12.115ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60452025-04-30T06:43:12.116ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d1 2a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .*..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53195 req_id = cd00e115-5430-4a0d-9c03-80943a549862
60462025-04-30T06:43:12.116ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53546,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53195 req_id = cd00e115-5430-4a0d-9c03-80943a549862
60472025-04-30T06:43:12.116ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
60482025-04-30T06:43:12.116ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60492025-04-30T06:43:12.116ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53195 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cd00e115-5430-4a0d-9c03-80943a549862
60502025-04-30T06:43:12.117ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4256, tv_nsec: 916391428 } })
60512025-04-30T06:43:12.117ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
60522025-04-30T06:43:12.118ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ce 5b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .[..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64395 req_id = 5dcb3573-1baa-4f1d-ac9f-d0aedec4f281
60532025-04-30T06:43:12.118ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52827,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64395 req_id = 5dcb3573-1baa-4f1d-ac9f-d0aedec4f281
60542025-04-30T06:43:12.118ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60552025-04-30T06:43:12.118ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64395 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5dcb3573-1baa-4f1d-ac9f-d0aedec4f281
60562025-04-30T06:43:12.184ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60572025-04-30T06:43:12.185ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60582025-04-30T06:43:12.185ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60592025-04-30T06:43:12.185ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60602025-04-30T06:43:12.185ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
60612025-04-30T06:43:12.186ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60622025-04-30T06:43:12.186ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: eb 4a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .J..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52683 req_id = 70171940-469b-4bb1-8686-d438ce469041
60632025-04-30T06:43:12.186ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60234,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52683 req_id = 70171940-469b-4bb1-8686-d438ce469041
60642025-04-30T06:43:12.187ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
60652025-04-30T06:43:12.187ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60234, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52683 req_id = 70171940-469b-4bb1-8686-d438ce469041
60662025-04-30T06:43:12.188ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
60672025-04-30T06:43:12.188ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
60682025-04-30T06:43:12.188ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
60692025-04-30T06:43:12.189ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60702025-04-30T06:43:12.194ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60712025-04-30T06:43:12.194ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60722025-04-30T06:43:12.194ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
60732025-04-30T06:43:12.195ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60742025-04-30T06:43:12.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c729909f-7a7b-4885-9c8b-db3cb6653a88", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60752025-04-30T06:43:12.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
60762025-04-30T06:43:12.199ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60772025-04-30T06:43:12.199ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
60782025-04-30T06:43:12.201ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60792025-04-30T06:43:12.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a70a1d3-39ad-4d4c-ae19-1c350c47d8c7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60802025-04-30T06:43:12.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
60812025-04-30T06:43:12.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c92b61f8-35e3-40d7-abee-88b3f131ef23", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60822025-04-30T06:43:12.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
60832025-04-30T06:43:12.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5c0acc7-5e33-4607-9944-d54d6484f3f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60842025-04-30T06:43:12.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
60852025-04-30T06:43:12.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7551fbd-a0be-4948-8a21-5df658e14b76", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60862025-04-30T06:43:12.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
60872025-04-30T06:43:12.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c63262e7-e58f-437a-bba2-35fe3b54a13b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60882025-04-30T06:43:12.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
60892025-04-30T06:43:12.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd142b3-b05f-4eab-a05d-a1a39cb4aea9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60902025-04-30T06:43:12.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
60912025-04-30T06:43:12.237ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
60922025-04-30T06:43:12.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12b78039-3fca-4ab5-8368-1c6293d012bb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60932025-04-30T06:43:12.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
60942025-04-30T06:43:12.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c757b59-1688-4380-b696-ff667f924279", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60952025-04-30T06:43:12.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
60962025-04-30T06:43:12.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a7ff6e1-875c-4bcc-a0a9-94632ca6e996", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60972025-04-30T06:43:12.243ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
60982025-04-30T06:43:12.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2dc00ab-6229-4f84-a30d-369b2f62b547", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
60992025-04-30T06:43:12.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
61002025-04-30T06:43:12.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d8bb509-0aed-4375-8d44-9c583cd4818f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61012025-04-30T06:43:12.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
61022025-04-30T06:43:12.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d2b0715-d5f5-4536-9f1b-d00bc4003701", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61032025-04-30T06:43:12.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
61042025-04-30T06:43:12.251ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d75a7b4e-9d18-43af-b7a4-03bf59f8fbf9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61052025-04-30T06:43:12.251ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
61062025-04-30T06:43:12.284ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0468e886-2fcf-4711-86eb-ab182f3487b7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61072025-04-30T06:43:12.284ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
61082025-04-30T06:43:12.292ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61092025-04-30T06:43:12.292ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61102025-04-30T06:43:12.292ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61112025-04-30T06:43:12.292ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61122025-04-30T06:43:12.293ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61132025-04-30T06:43:12.296ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61142025-04-30T06:43:12.304ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61152025-04-30T06:43:12.304ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61162025-04-30T06:43:12.304ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61172025-04-30T06:43:12.305ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61182025-04-30T06:43:12.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad6c89a0-1aa4-4ae7-9d92-2f45264b2440", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61192025-04-30T06:43:12.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
61202025-04-30T06:43:12.309ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61212025-04-30T06:43:12.309ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61222025-04-30T06:43:12.311ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61232025-04-30T06:43:12.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b276caf6-66ba-48da-9631-979d98cf0604", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61242025-04-30T06:43:12.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
61252025-04-30T06:43:12.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f380aba8-59b0-472f-901a-35c8ee6feb86", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61262025-04-30T06:43:12.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
61272025-04-30T06:43:12.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3494092-b937-4fa4-b086-30ed572aa5d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61282025-04-30T06:43:12.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
61292025-04-30T06:43:12.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36516eac-68dd-49fb-a9b1-3518e1855602", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61302025-04-30T06:43:12.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
61312025-04-30T06:43:12.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "453766a7-01c8-47f2-821c-d9f89cf1f643", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61322025-04-30T06:43:12.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
61332025-04-30T06:43:12.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d9584d7-9dc9-41c1-b36e-a9bd0b6e37ce", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61342025-04-30T06:43:12.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
61352025-04-30T06:43:12.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c6fa619-dc42-4ad4-83e0-019ffd7f5ad7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61362025-04-30T06:43:12.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
61372025-04-30T06:43:12.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2a53b8-78de-46ac-924a-3f18b3a15744", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61382025-04-30T06:43:12.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
61392025-04-30T06:43:12.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e61b8e8a-34d8-4cca-8770-18f4088bd478", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61402025-04-30T06:43:12.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
61412025-04-30T06:43:12.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7fc8c7-db50-4e5c-aae3-baa2f42e72da", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61422025-04-30T06:43:12.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
61432025-04-30T06:43:12.348ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
61442025-04-30T06:43:12.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "707b2f33-eea1-43e0-bf5f-a97ef9db59cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61452025-04-30T06:43:12.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
61462025-04-30T06:43:12.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55991e18-d643-4640-8906-4244cee3bb69", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61472025-04-30T06:43:12.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
61482025-04-30T06:43:12.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee9c76c0-bc66-409c-9290-8c157a3828cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61492025-04-30T06:43:12.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
61502025-04-30T06:43:12.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c4b0827-9769-44f8-9f43-3ae2346d70dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61512025-04-30T06:43:12.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
61522025-04-30T06:43:12.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc0a1791-c6b7-451f-8dcf-2d0e7da82ba8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61532025-04-30T06:43:12.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
61542025-04-30T06:43:12.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa34d1c-089a-48b7-aa83-ac64c7b419b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61552025-04-30T06:43:12.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
61562025-04-30T06:43:12.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48d6617-4721-41e2-88e9-60f51204df91", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61572025-04-30T06:43:12.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
61582025-04-30T06:43:12.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdd09a79-291f-4b2f-a421-9e5d906cc36a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61592025-04-30T06:43:12.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
61602025-04-30T06:43:12.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f7b8fd0-07c5-4dbc-a18a-81ef613fc8a0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61612025-04-30T06:43:12.374ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
61622025-04-30T06:43:12.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "445a66e0-6e35-42b9-b6e1-1dc5e1387cd8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61632025-04-30T06:43:12.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
61642025-04-30T06:43:12.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c00d5ca-69bb-487e-8add-4dbc85a8b143", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61652025-04-30T06:43:12.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
61662025-04-30T06:43:12.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6617b8df-e16e-4c18-9c3d-10dbf481c784", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61672025-04-30T06:43:12.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
61682025-04-30T06:43:12.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65d5fa65-2f3e-4942-9027-98d76cc15b24", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61692025-04-30T06:43:12.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
61702025-04-30T06:43:12.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d707ad9e-5dfd-412f-8824-d93bfdf258c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61712025-04-30T06:43:12.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
61722025-04-30T06:43:12.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e7f1e0c-29ca-4225-a391-6058aa4a94cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61732025-04-30T06:43:12.403ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
61742025-04-30T06:43:12.404ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61752025-04-30T06:43:12.404ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61762025-04-30T06:43:12.404ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61772025-04-30T06:43:12.404ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61782025-04-30T06:43:12.404ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61792025-04-30T06:43:12.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8253f428-a431-4329-87e1-861b7faccc19", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61802025-04-30T06:43:12.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
61812025-04-30T06:43:12.408ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61822025-04-30T06:43:12.414ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61832025-04-30T06:43:12.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61842025-04-30T06:43:12.415ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61852025-04-30T06:43:12.415ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61862025-04-30T06:43:12.416ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b1fdac-8048-4106-850e-e83333cf8fcc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61872025-04-30T06:43:12.417ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
61882025-04-30T06:43:12.418ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61892025-04-30T06:43:12.418ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
61902025-04-30T06:43:12.420ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61912025-04-30T06:43:12.421ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f724ff5-2fc6-472f-8852-d72a5b4975cd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61922025-04-30T06:43:12.421ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
61932025-04-30T06:43:12.422ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa5365bd-7992-43f7-8cc4-231fd8d76e31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
61942025-04-30T06:43:12.423ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61952025-04-30T06:43:12.423ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61962025-04-30T06:43:12.423ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61972025-04-30T06:43:12.423ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61982025-04-30T06:43:12.423ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61992025-04-30T06:43:12.432ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62002025-04-30T06:43:12.442ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62012025-04-30T06:43:12.442ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
62022025-04-30T06:43:12.442ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62032025-04-30T06:43:12.442ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62042025-04-30T06:43:12.442ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62052025-04-30T06:43:12.443ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62062025-04-30T06:43:12.449ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62072025-04-30T06:43:12.455ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62082025-04-30T06:43:12.455ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62092025-04-30T06:43:12.455ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62102025-04-30T06:43:12.456ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62112025-04-30T06:43:12.551ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
62122025-04-30T06:43:12.551ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
62132025-04-30T06:43:12.552ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:49729 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
62142025-04-30T06:43:12.553ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:49729 req_id = f644c853-aa10-4b45-b3ea-23739927e4f8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62152025-04-30T06:43:12.554ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 861 local_addr = [::1]:45714 method = GET remote_addr = [::1]:49729 req_id = f644c853-aa10-4b45-b3ea-23739927e4f8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62162025-04-30T06:43:12.555ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f644c853-aa10-4b45-b3ea-23739927e4f8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62172025-04-30T06:43:12.561ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62182025-04-30T06:43:12.561ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62192025-04-30T06:43:12.561ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
62202025-04-30T06:43:12.563ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62212025-04-30T06:43:12.563ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62222025-04-30T06:43:12.563ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62232025-04-30T06:43:12.563ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62242025-04-30T06:43:12.568ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62252025-04-30T06:43:12.572ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62262025-04-30T06:43:12.579ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62272025-04-30T06:43:12.579ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62282025-04-30T06:43:12.579ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62292025-04-30T06:43:12.580ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62302025-04-30T06:43:12.595ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
62312025-04-30T06:43:12.618ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62322025-04-30T06:43:12.618ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62332025-04-30T06:43:12.618ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 523.499749ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62342025-04-30T06:43:12.619ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62352025-04-30T06:43:12.619ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62362025-04-30T06:43:12.619ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62372025-04-30T06:43:12.619ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62382025-04-30T06:43:12.619ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62392025-04-30T06:43:12.622ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62402025-04-30T06:43:12.630ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62412025-04-30T06:43:12.643ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62422025-04-30T06:43:12.643ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62432025-04-30T06:43:12.643ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62442025-04-30T06:43:12.644ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62452025-04-30T06:43:12.647ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62462025-04-30T06:43:12.648ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62472025-04-30T06:43:12.648ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62482025-04-30T06:43:12.648ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
62492025-04-30T06:43:12.649ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62502025-04-30T06:43:12.649ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
62512025-04-30T06:43:12.649ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62522025-04-30T06:43:12.650ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 69 31 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 i1..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56680 req_id = ede1fd97-26cd-4e7c-9c0a-fdc93bd6bb2e
62532025-04-30T06:43:12.650ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26929,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56680 req_id = ede1fd97-26cd-4e7c-9c0a-fdc93bd6bb2e
62542025-04-30T06:43:12.650ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
62552025-04-30T06:43:12.650ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62562025-04-30T06:43:12.650ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56680 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ede1fd97-26cd-4e7c-9c0a-fdc93bd6bb2e
62572025-04-30T06:43:12.651ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 450222923 } })
62582025-04-30T06:43:12.651ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
62592025-04-30T06:43:12.652ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62602025-04-30T06:43:12.657ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62612025-04-30T06:43:12.657ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62622025-04-30T06:43:12.657ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
62632025-04-30T06:43:12.657ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62642025-04-30T06:43:12.658ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e9 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63232 req_id = 9188c72b-b714-418d-898c-d7cb8f485c8a
62652025-04-30T06:43:12.658ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59743,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63232 req_id = 9188c72b-b714-418d-898c-d7cb8f485c8a
62662025-04-30T06:43:12.658ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62672025-04-30T06:43:12.658ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63232 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9188c72b-b714-418d-898c-d7cb8f485c8a
62682025-04-30T06:43:12.716ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62692025-04-30T06:43:12.717ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62702025-04-30T06:43:12.717ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62712025-04-30T06:43:12.718ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d3 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53328 req_id = 5a369b88-b30f-41c1-86bd-ab45ea6be605
62722025-04-30T06:43:12.718ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54034,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53328 req_id = 5a369b88-b30f-41c1-86bd-ab45ea6be605
62732025-04-30T06:43:12.718ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
62742025-04-30T06:43:12.718ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54034, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53328 req_id = 5a369b88-b30f-41c1-86bd-ab45ea6be605
62752025-04-30T06:43:12.722ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62762025-04-30T06:43:12.722ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
62772025-04-30T06:43:12.723ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62782025-04-30T06:43:12.723ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
62792025-04-30T06:43:12.723ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
62802025-04-30T06:43:12.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
62812025-04-30T06:43:12.724ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62822025-04-30T06:43:12.724ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62832025-04-30T06:43:12.724ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62842025-04-30T06:43:12.724ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62852025-04-30T06:43:12.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "318be16b-bce6-4f24-8464-b193f9a268c9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62862025-04-30T06:43:12.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
62872025-04-30T06:43:12.728ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f53250-6445-4324-a180-bab89c260f22", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62882025-04-30T06:43:12.728ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
62892025-04-30T06:43:12.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8969d522-9c16-4537-8999-dddfe9e11400", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62902025-04-30T06:43:12.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
62912025-04-30T06:43:12.737ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea9d472-fb27-4764-943c-77a8ca645516", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62922025-04-30T06:43:12.737ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
62932025-04-30T06:43:12.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee77d643-cba3-4597-ac98-4194961919b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62942025-04-30T06:43:12.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
62952025-04-30T06:43:12.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73b8b56a-fcc3-452e-99b2-1fa3420a16a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62962025-04-30T06:43:12.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
62972025-04-30T06:43:12.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2624b15-f3f2-434e-85ac-42f5350b998e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
62982025-04-30T06:43:12.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
62992025-04-30T06:43:12.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "246452f2-4dd2-4925-885c-f5f9ecc4dae1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63002025-04-30T06:43:12.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
63012025-04-30T06:43:12.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edfb0799-508f-4649-af0f-ba23284ba6ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63022025-04-30T06:43:12.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
63032025-04-30T06:43:12.757ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
63042025-04-30T06:43:12.758ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25c1ef24-32b9-43e8-ba73-dff485234733", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63052025-04-30T06:43:12.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
63062025-04-30T06:43:12.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89f8d7b5-5588-4aff-86e4-2f73a9c0f706", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63072025-04-30T06:43:12.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
63082025-04-30T06:43:12.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aed4496-e67b-4f32-9b73-2b4e3d433169", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63092025-04-30T06:43:12.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
63102025-04-30T06:43:12.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4fb9ba-8bf3-4207-befa-d1b12706c6e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63112025-04-30T06:43:12.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
63122025-04-30T06:43:12.772ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42c4d251-bf64-4d2e-bb32-5f9d530e216b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63132025-04-30T06:43:12.772ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
63142025-04-30T06:43:12.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6524e862-af23-48c3-92ed-ff9fc3ba939f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63152025-04-30T06:43:12.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
63162025-04-30T06:43:12.778ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5b14c9e-33c3-4532-ac84-981e11f83099", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63172025-04-30T06:43:12.778ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
63182025-04-30T06:43:12.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7792a5a1-00fa-4a63-94b5-2bf6b45800a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63192025-04-30T06:43:12.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
63202025-04-30T06:43:12.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ffd0c8-600a-4016-b86e-291e20df8dc5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63212025-04-30T06:43:12.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
63222025-04-30T06:43:12.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "623b14cf-381b-4789-a243-41fce7b30ef8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63232025-04-30T06:43:12.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
63242025-04-30T06:43:12.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c75a2374-91c9-48c3-a65f-3dc8c8ec4bc4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63252025-04-30T06:43:12.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
63262025-04-30T06:43:12.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01907c32-53c8-4f2a-9c11-4a06c9cd07be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63272025-04-30T06:43:12.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
63282025-04-30T06:43:12.790ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88eadc31-da13-478e-af8b-80e1f67a4e4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63292025-04-30T06:43:12.790ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
63302025-04-30T06:43:12.791ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32c267c3-419a-43a7-9523-a5660e74d000", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63312025-04-30T06:43:12.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
63322025-04-30T06:43:12.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5650273f-e124-41ed-8474-f1d8fc26e62b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63332025-04-30T06:43:12.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
63342025-04-30T06:43:12.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb2b973-8332-47b5-87e6-8fb767ba1ede", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63352025-04-30T06:43:12.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
63362025-04-30T06:43:12.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "179e385e-1366-4e08-9205-fdab10d960cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63372025-04-30T06:43:12.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
63382025-04-30T06:43:12.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35de51ad-d1d3-4db6-ac9d-2f60153365bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63392025-04-30T06:43:12.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
63402025-04-30T06:43:12.805ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "354a76d7-8795-44f9-99fb-ab1087398810", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63412025-04-30T06:43:12.805ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
63422025-04-30T06:43:12.807ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63432025-04-30T06:43:12.807ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63442025-04-30T06:43:12.808ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63452025-04-30T06:43:12.808ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
63462025-04-30T06:43:12.808ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63472025-04-30T06:43:12.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6563553-f383-4090-8c22-a3df031a9af5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63482025-04-30T06:43:12.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
63492025-04-30T06:43:12.811ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63502025-04-30T06:43:12.816ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63512025-04-30T06:43:12.816ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63522025-04-30T06:43:12.816ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
63532025-04-30T06:43:12.817ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63542025-04-30T06:43:12.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6cfd45f-00ad-4629-9d85-b8c740085600", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63552025-04-30T06:43:12.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
63562025-04-30T06:43:12.819ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63572025-04-30T06:43:12.819ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
63582025-04-30T06:43:12.819ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63592025-04-30T06:43:12.821ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7841dd7e-02ab-46d3-b07a-ed6ea918f958", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63602025-04-30T06:43:12.821ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
63612025-04-30T06:43:12.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5104e27-3417-4618-b44b-e08a7aceaeb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63622025-04-30T06:43:12.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
63632025-04-30T06:43:12.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1d94a2-e3dd-4032-b7ad-2a9d635b2dd7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63642025-04-30T06:43:12.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
63652025-04-30T06:43:12.826ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85fff3e6-17c4-4b38-8539-65ba0afc7d3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63662025-04-30T06:43:12.826ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
63672025-04-30T06:43:12.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cb5dfec-43b8-4604-b8a7-49714b808425", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63682025-04-30T06:43:12.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
63692025-04-30T06:43:12.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f76ee11-5d89-4c56-b492-c26154ff5f76", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63702025-04-30T06:43:12.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
63712025-04-30T06:43:12.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9e19b12-39f2-4e12-a1b6-3cfaf6b9b23c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63722025-04-30T06:43:12.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
63732025-04-30T06:43:12.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b955fe0a-1b30-4e90-822b-6d5f3082cf2e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63742025-04-30T06:43:12.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
63752025-04-30T06:43:12.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb91c91-7214-4906-bbd8-80c26a6a33a8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63762025-04-30T06:43:12.844ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
63772025-04-30T06:43:12.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "497d81d1-871e-444a-999f-13757e018e9c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63782025-04-30T06:43:12.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
63792025-04-30T06:43:12.851ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
63802025-04-30T06:43:12.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "575d3208-e940-443b-a23a-2e0dbb7687b4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63812025-04-30T06:43:12.855ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
63822025-04-30T06:43:12.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5520ac4-a37a-437e-b39b-380656cbdbe8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63832025-04-30T06:43:12.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
63842025-04-30T06:43:12.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67254536-d8c1-4bfd-af2a-2bf6f8b9df01", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63852025-04-30T06:43:12.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
63862025-04-30T06:43:12.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb5ddf31-7454-4522-b3d9-c720b13386bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63872025-04-30T06:43:12.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
63882025-04-30T06:43:12.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d4ae09-9a3a-44a6-b119-ed19c2909f2c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
63892025-04-30T06:43:12.861ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63902025-04-30T06:43:12.861ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63912025-04-30T06:43:12.861ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63922025-04-30T06:43:12.861ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63932025-04-30T06:43:12.862ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63942025-04-30T06:43:12.866ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63952025-04-30T06:43:12.873ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63962025-04-30T06:43:12.873ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63972025-04-30T06:43:12.873ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63982025-04-30T06:43:12.873ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63992025-04-30T06:43:12.873ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64002025-04-30T06:43:12.873ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64012025-04-30T06:43:12.876ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64022025-04-30T06:43:12.880ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64032025-04-30T06:43:12.880ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64042025-04-30T06:43:12.880ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64052025-04-30T06:43:12.881ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64062025-04-30T06:43:12.936ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
64072025-04-30T06:43:12.937ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
64082025-04-30T06:43:12.937ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64092025-04-30T06:43:12.937ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64102025-04-30T06:43:12.937ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64112025-04-30T06:43:12.937ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64122025-04-30T06:43:12.938ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64132025-04-30T06:43:12.938ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:44321 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64142025-04-30T06:43:12.939ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:44321 req_id = 82f0ed96-18c4-41ca-b53f-cbc82922c614 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64152025-04-30T06:43:12.940ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 601 local_addr = [::1]:45714 method = GET remote_addr = [::1]:44321 req_id = 82f0ed96-18c4-41ca-b53f-cbc82922c614 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64162025-04-30T06:43:12.940ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "82f0ed96-18c4-41ca-b53f-cbc82922c614", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:11 GMT"} })
64172025-04-30T06:43:12.943ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64182025-04-30T06:43:12.943ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64192025-04-30T06:43:12.943ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
64202025-04-30T06:43:12.945ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64212025-04-30T06:43:12.945ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64222025-04-30T06:43:12.945ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64232025-04-30T06:43:12.946ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64242025-04-30T06:43:12.946ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64252025-04-30T06:43:12.947ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64262025-04-30T06:43:12.951ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64272025-04-30T06:43:12.951ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64282025-04-30T06:43:12.951ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64292025-04-30T06:43:12.952ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64302025-04-30T06:43:12.963ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64312025-04-30T06:43:12.968ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64322025-04-30T06:43:12.968ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64332025-04-30T06:43:12.968ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64342025-04-30T06:43:12.969ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64352025-04-30T06:43:12.972ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64362025-04-30T06:43:12.972ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64372025-04-30T06:43:12.973ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64382025-04-30T06:43:12.976ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64392025-04-30T06:43:12.976ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64402025-04-30T06:43:12.976ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
64412025-04-30T06:43:12.977ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
64422025-04-30T06:43:12.983ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
64432025-04-30T06:43:12.983ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
64442025-04-30T06:43:12.989ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
64452025-04-30T06:43:12.989ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
64462025-04-30T06:43:13.013ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64472025-04-30T06:43:13.013ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64482025-04-30T06:43:13.013ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 394.35717ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64492025-04-30T06:43:13.013ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64502025-04-30T06:43:13.013ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64512025-04-30T06:43:13.013ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64522025-04-30T06:43:13.013ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64532025-04-30T06:43:13.013ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64542025-04-30T06:43:13.014ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64552025-04-30T06:43:13.016ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64562025-04-30T06:43:13.021ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64572025-04-30T06:43:13.021ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64582025-04-30T06:43:13.021ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64592025-04-30T06:43:13.021ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64602025-04-30T06:43:13.025ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
64612025-04-30T06:43:13.025ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64622025-04-30T06:43:13.027ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
64632025-04-30T06:43:13.029ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d4 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55111 req_id = eaa63c3a-c294-4740-bdbe-28d2559fc008
64642025-04-30T06:43:13.029ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54346,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55111 req_id = eaa63c3a-c294-4740-bdbe-28d2559fc008
64652025-04-30T06:43:13.029ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
64662025-04-30T06:43:13.029ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64672025-04-30T06:43:13.030ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55111 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = eaa63c3a-c294-4740-bdbe-28d2559fc008
64682025-04-30T06:43:13.030ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4257, tv_nsec: 829596419 } })
64692025-04-30T06:43:13.030ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
64702025-04-30T06:43:13.031ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 60 2c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e `,..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49519 req_id = 6e619e87-26f4-4762-87e7-00f62c8c1c6a
64712025-04-30T06:43:13.031ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24620,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49519 req_id = 6e619e87-26f4-4762-87e7-00f62c8c1c6a
64722025-04-30T06:43:13.032ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64732025-04-30T06:43:13.032ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49519 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6e619e87-26f4-4762-87e7-00f62c8c1c6a
64742025-04-30T06:43:13.108ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64752025-04-30T06:43:13.109ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64762025-04-30T06:43:13.109ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64772025-04-30T06:43:13.109ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64782025-04-30T06:43:13.109ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64792025-04-30T06:43:13.110ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64802025-04-30T06:43:13.110ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
64812025-04-30T06:43:13.111ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: eb 9e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59379 req_id = f5d29f14-4755-49a2-8fd3-09de84defdde
64822025-04-30T06:43:13.111ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60318,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59379 req_id = f5d29f14-4755-49a2-8fd3-09de84defdde
64832025-04-30T06:43:13.111ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
64842025-04-30T06:43:13.111ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60318, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59379 req_id = f5d29f14-4755-49a2-8fd3-09de84defdde
64852025-04-30T06:43:13.113ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
64862025-04-30T06:43:13.113ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
64872025-04-30T06:43:13.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
64882025-04-30T06:43:13.113ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64892025-04-30T06:43:13.120ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64902025-04-30T06:43:13.120ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64912025-04-30T06:43:13.120ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64922025-04-30T06:43:13.121ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64932025-04-30T06:43:13.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3167cc6-067b-452b-9590-0511422a9849", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
64942025-04-30T06:43:13.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
64952025-04-30T06:43:13.124ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64962025-04-30T06:43:13.125ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
64972025-04-30T06:43:13.125ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64982025-04-30T06:43:13.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f535e57-a8e1-47d4-9176-18ba46bd7562", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
64992025-04-30T06:43:13.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
65002025-04-30T06:43:13.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "931264de-ba9c-4934-834e-9ac9c665bf95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65012025-04-30T06:43:13.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
65022025-04-30T06:43:13.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35580622-a4b0-4108-a164-856a37223704", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65032025-04-30T06:43:13.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
65042025-04-30T06:43:13.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6444739f-e021-4e43-a748-cf175ce426eb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65052025-04-30T06:43:13.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
65062025-04-30T06:43:13.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2379797d-0db7-406e-bb8e-e0b428c5c55f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65072025-04-30T06:43:13.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
65082025-04-30T06:43:13.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5bec3da-4a5a-417a-856c-70d78eabbf07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65092025-04-30T06:43:13.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
65102025-04-30T06:43:13.157ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
65112025-04-30T06:43:13.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ac5efec-45a4-4630-8310-9c5af845b51d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65122025-04-30T06:43:13.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
65132025-04-30T06:43:13.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d904027a-f696-40b0-8c0d-c88217f098e0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65142025-04-30T06:43:13.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
65152025-04-30T06:43:13.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e29d9b03-a61b-4b68-9cb9-abd86a5d55ef", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65162025-04-30T06:43:13.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
65172025-04-30T06:43:13.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67490013-91c9-4a50-b669-17494ea8562a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65182025-04-30T06:43:13.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
65192025-04-30T06:43:13.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a12e01-7897-4365-914b-c858eaeb1946", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65202025-04-30T06:43:13.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
65212025-04-30T06:43:13.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa1e48f1-35e7-4057-b621-f95e84e40555", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65222025-04-30T06:43:13.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
65232025-04-30T06:43:13.165ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24066d8f-b0fd-4df1-8e0b-4e80e4c35481", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65242025-04-30T06:43:13.165ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
65252025-04-30T06:43:13.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35095e52-1bd5-4d1d-83f3-6dc9e28a28dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65262025-04-30T06:43:13.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
65272025-04-30T06:43:13.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16ea8887-33d9-4dd0-bea8-076bb81aa9f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65282025-04-30T06:43:13.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
65292025-04-30T06:43:13.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf46c29-b01b-4fc6-8923-534590dce5b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65302025-04-30T06:43:13.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
65312025-04-30T06:43:13.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8325bdd0-0632-48be-a6e8-1ff864725f81", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65322025-04-30T06:43:13.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
65332025-04-30T06:43:13.179ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ddfb230-677d-4c8a-9564-84021c7ba640", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65342025-04-30T06:43:13.179ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
65352025-04-30T06:43:13.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e63304a9-c306-4d42-abde-b26893f8da6c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65362025-04-30T06:43:13.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
65372025-04-30T06:43:13.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0337eb9-ec77-4949-b8de-621f68d89458", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65382025-04-30T06:43:13.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
65392025-04-30T06:43:13.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02390232-e8d2-464b-b42b-b65a9ed7fa79", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65402025-04-30T06:43:13.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
65412025-04-30T06:43:13.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d7309a-3f1d-4bfa-850a-ff619a52d2ff", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65422025-04-30T06:43:13.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
65432025-04-30T06:43:13.187ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb150b78-118f-4a8a-9255-c67f7c5113bc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65442025-04-30T06:43:13.187ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
65452025-04-30T06:43:13.190ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "164bd68f-9c57-482c-b750-474b3e200891", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65462025-04-30T06:43:13.190ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
65472025-04-30T06:43:13.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49624bc7-94a1-4662-8a56-acc3034b075e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65482025-04-30T06:43:13.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
65492025-04-30T06:43:13.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccf77700-81dd-4dfe-8134-ae99d437e123", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65502025-04-30T06:43:13.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
65512025-04-30T06:43:13.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1ccbed6-9e0f-4244-910b-ef654c6b6db9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65522025-04-30T06:43:13.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
65532025-04-30T06:43:13.203ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96fc3fe3-1b73-4357-b62e-10bd1da965bf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65542025-04-30T06:43:13.203ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
65552025-04-30T06:43:13.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd8ce1b2-0088-4357-a59c-5babbeae2805", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65562025-04-30T06:43:13.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
65572025-04-30T06:43:13.208ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65582025-04-30T06:43:13.208ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65592025-04-30T06:43:13.208ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65602025-04-30T06:43:13.208ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
65612025-04-30T06:43:13.211ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65622025-04-30T06:43:13.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdb0e992-88ea-46fa-bf88-c281420d6595", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65632025-04-30T06:43:13.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
65642025-04-30T06:43:13.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2665de8c-f415-4182-a979-8cd95f3b2386", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65652025-04-30T06:43:13.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
65662025-04-30T06:43:13.214ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65672025-04-30T06:43:13.219ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65682025-04-30T06:43:13.219ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65692025-04-30T06:43:13.220ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
65702025-04-30T06:43:13.220ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65712025-04-30T06:43:13.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b6cf0d0-19df-4a21-8bfd-7c9c8688aa1b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65722025-04-30T06:43:13.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
65732025-04-30T06:43:13.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "772ef6d0-f119-4194-84a2-d1330cc57510", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65742025-04-30T06:43:13.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
65752025-04-30T06:43:13.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3db043f-f37d-4cab-b93a-709a3d366887", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65762025-04-30T06:43:13.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
65772025-04-30T06:43:13.223ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65782025-04-30T06:43:13.224ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
65792025-04-30T06:43:13.225ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65802025-04-30T06:43:13.228ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d236b4dd-d42f-425f-93fb-f6e2aa337daa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65812025-04-30T06:43:13.228ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
65822025-04-30T06:43:13.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d31bf7-d524-4b99-a911-5b00db121824", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65832025-04-30T06:43:13.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
65842025-04-30T06:43:13.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "482f2cfe-7c17-448c-aa81-d9e55ab6b331", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65852025-04-30T06:43:13.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
65862025-04-30T06:43:13.236ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc274e1-9039-4072-aabb-c6b4f0bbb96e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65872025-04-30T06:43:13.236ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
65882025-04-30T06:43:13.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b900a7b7-5993-457f-b12f-77c2478a6141", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65892025-04-30T06:43:13.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
65902025-04-30T06:43:13.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c132db01-b010-405c-b3fe-aee9990ecd05", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65912025-04-30T06:43:13.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
65922025-04-30T06:43:13.243ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed777d7-f0c1-48cf-9df8-49fd212dc42b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65932025-04-30T06:43:13.243ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
65942025-04-30T06:43:13.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d4648b3-6a49-4488-8df1-7e8f7bd2d155", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65952025-04-30T06:43:13.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
65962025-04-30T06:43:13.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d2b3c8-4c24-4ca0-b8a5-4a544dbdc661", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65972025-04-30T06:43:13.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
65982025-04-30T06:43:13.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a1234ce-1deb-4875-90c7-8f38476c3a27", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
65992025-04-30T06:43:13.248ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66002025-04-30T06:43:13.248ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66012025-04-30T06:43:13.248ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66022025-04-30T06:43:13.248ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66032025-04-30T06:43:13.248ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66042025-04-30T06:43:13.252ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66052025-04-30T06:43:13.257ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66062025-04-30T06:43:13.257ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66072025-04-30T06:43:13.257ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66082025-04-30T06:43:13.257ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66092025-04-30T06:43:13.257ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66102025-04-30T06:43:13.257ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66112025-04-30T06:43:13.261ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66122025-04-30T06:43:13.266ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66132025-04-30T06:43:13.266ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66142025-04-30T06:43:13.266ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66152025-04-30T06:43:13.268ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66162025-04-30T06:43:13.324ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
66172025-04-30T06:43:13.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
66182025-04-30T06:43:13.327ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:61170 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66192025-04-30T06:43:13.335ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
66202025-04-30T06:43:13.335ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:61170 req_id = 3dfaa98f-ff99-49a2-ac00-374353b75e62 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66212025-04-30T06:43:13.336ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 883 local_addr = [::1]:45714 method = GET remote_addr = [::1]:61170 req_id = 3dfaa98f-ff99-49a2-ac00-374353b75e62 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66222025-04-30T06:43:13.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dfaa98f-ff99-49a2-ac00-374353b75e62", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66232025-04-30T06:43:13.345ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66242025-04-30T06:43:13.345ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66252025-04-30T06:43:13.345ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
66262025-04-30T06:43:13.349ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66272025-04-30T06:43:13.349ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66282025-04-30T06:43:13.349ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66292025-04-30T06:43:13.349ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66302025-04-30T06:43:13.349ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66312025-04-30T06:43:13.357ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66322025-04-30T06:43:13.361ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66332025-04-30T06:43:13.361ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66342025-04-30T06:43:13.362ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66352025-04-30T06:43:13.362ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66362025-04-30T06:43:13.387ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66372025-04-30T06:43:13.387ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66382025-04-30T06:43:13.387ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66392025-04-30T06:43:13.387ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
66402025-04-30T06:43:13.389ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66412025-04-30T06:43:13.395ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66422025-04-30T06:43:13.400ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66432025-04-30T06:43:13.401ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66442025-04-30T06:43:13.401ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
66452025-04-30T06:43:13.401ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66462025-04-30T06:43:13.423ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66472025-04-30T06:43:13.423ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
66482025-04-30T06:43:13.432ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66492025-04-30T06:43:13.433ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66502025-04-30T06:43:13.433ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66512025-04-30T06:43:13.433ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 420.236183ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66522025-04-30T06:43:13.433ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66532025-04-30T06:43:13.433ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66542025-04-30T06:43:13.434ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66552025-04-30T06:43:13.435ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66562025-04-30T06:43:13.435ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66572025-04-30T06:43:13.436ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66582025-04-30T06:43:13.441ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66592025-04-30T06:43:13.447ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66602025-04-30T06:43:13.448ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66612025-04-30T06:43:13.448ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66622025-04-30T06:43:13.448ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66632025-04-30T06:43:13.451ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
66642025-04-30T06:43:13.451ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66652025-04-30T06:43:13.453ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0c 74 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .t..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61251 req_id = e635e228-7f26-4dcb-9f5e-e5fa8a700040
66662025-04-30T06:43:13.453ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3188,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61251 req_id = e635e228-7f26-4dcb-9f5e-e5fa8a700040
66672025-04-30T06:43:13.453ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
66682025-04-30T06:43:13.454ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66692025-04-30T06:43:13.454ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61251 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e635e228-7f26-4dcb-9f5e-e5fa8a700040
66702025-04-30T06:43:13.455ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 253948553 } })
66712025-04-30T06:43:13.455ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
66722025-04-30T06:43:13.456ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bf 48 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .H..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53785 req_id = 06a8ea75-7c6c-4a09-ad8f-08ff73c9eb7b
66732025-04-30T06:43:13.456ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48968,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53785 req_id = 06a8ea75-7c6c-4a09-ad8f-08ff73c9eb7b
66742025-04-30T06:43:13.456ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66752025-04-30T06:43:13.456ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53785 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 06a8ea75-7c6c-4a09-ad8f-08ff73c9eb7b
66762025-04-30T06:43:13.511ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66772025-04-30T06:43:13.511ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6b 26 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c k&..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65173 req_id = d2f07cd9-2f74-4cf2-91ec-5edaa5a3664e
66782025-04-30T06:43:13.511ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27430,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65173 req_id = d2f07cd9-2f74-4cf2-91ec-5edaa5a3664e
66792025-04-30T06:43:13.512ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
66802025-04-30T06:43:13.512ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27430, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65173 req_id = d2f07cd9-2f74-4cf2-91ec-5edaa5a3664e
66812025-04-30T06:43:13.513ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
66822025-04-30T06:43:13.513ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
66832025-04-30T06:43:13.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
66842025-04-30T06:43:13.516ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f725086-08c0-44a6-92f6-a2b5dd213941", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66852025-04-30T06:43:13.516ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
66862025-04-30T06:43:13.518ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e602362a-d92e-469b-a352-452e523f5e39", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66872025-04-30T06:43:13.519ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
66882025-04-30T06:43:13.522ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efde709e-3163-4b41-b0a8-446f9fa9576b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66892025-04-30T06:43:13.522ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
66902025-04-30T06:43:13.524ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0da6c7d3-270c-419c-9117-005b7c56647b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66912025-04-30T06:43:13.525ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
66922025-04-30T06:43:13.528ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aee6ed1-d8ad-47a9-a60b-26203c161316", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66932025-04-30T06:43:13.528ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
66942025-04-30T06:43:13.530ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d0e4c9-0116-4183-8f88-70fb9d422afa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66952025-04-30T06:43:13.530ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
66962025-04-30T06:43:13.540ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
66972025-04-30T06:43:13.542ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed23a9b5-4d2f-46b7-a17f-c1d09b566c64", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
66982025-04-30T06:43:13.542ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
66992025-04-30T06:43:13.543ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc6ead9-5bb0-4a95-ad8b-3c53920a1385", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67002025-04-30T06:43:13.544ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
67012025-04-30T06:43:13.545ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b52adda-0da4-485d-a90f-99f8a6fa10a6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67022025-04-30T06:43:13.545ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
67032025-04-30T06:43:13.545ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba2581fb-5bf3-4e1f-a1c8-3c4c19a5eefd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67042025-04-30T06:43:13.545ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
67052025-04-30T06:43:13.546ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a71afcbd-e35c-4a43-813d-db80bbcc21a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67062025-04-30T06:43:13.546ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
67072025-04-30T06:43:13.548ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38191a77-c30a-4724-a29f-2b1fc641fb4d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67082025-04-30T06:43:13.548ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
67092025-04-30T06:43:13.549ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08c2a9ed-7304-46b5-a9c9-8c4409cbea0e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67102025-04-30T06:43:13.549ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
67112025-04-30T06:43:13.551ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac6b03e7-1206-4214-994f-61b8ad2ca2fc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67122025-04-30T06:43:13.551ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
67132025-04-30T06:43:13.553ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11d826e5-8f44-4152-bb27-203fe58e1fbe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67142025-04-30T06:43:13.553ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
67152025-04-30T06:43:13.555ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6667c972-b67b-4835-8dae-ab2343821e2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67162025-04-30T06:43:13.556ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
67172025-04-30T06:43:13.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "036787ee-90a7-4cfa-b841-cbb482021223", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67182025-04-30T06:43:13.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
67192025-04-30T06:43:13.561ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a16bcee2-8669-48e0-90c6-514a16ebdaaf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67202025-04-30T06:43:13.561ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
67212025-04-30T06:43:13.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b98ec28c-e3c4-45c9-9f18-9130263aeba1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67222025-04-30T06:43:13.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
67232025-04-30T06:43:13.563ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d54e9fa-8b80-4395-bb6e-03f0d5e6503d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67242025-04-30T06:43:13.563ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
67252025-04-30T06:43:13.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fa067db-417d-4113-9f99-d261c3fd21ab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67262025-04-30T06:43:13.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
67272025-04-30T06:43:13.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef61cfb2-91ca-402c-883b-47d60cb8e638", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67282025-04-30T06:43:13.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
67292025-04-30T06:43:13.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "081437bd-a075-4fef-aa90-1003c7beec25", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67302025-04-30T06:43:13.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
67312025-04-30T06:43:13.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "026dc0a9-818f-4507-8356-e288a46b4db6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67322025-04-30T06:43:13.569ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
67332025-04-30T06:43:13.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d28ee9-c8dc-497c-8bbe-f27b560164a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67342025-04-30T06:43:13.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
67352025-04-30T06:43:13.586ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d30b6f62-1d1c-4d1b-a1cf-de0ff358c983", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67362025-04-30T06:43:13.587ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
67372025-04-30T06:43:13.591ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "840d4097-136b-4504-a48c-997bee07b58d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67382025-04-30T06:43:13.591ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
67392025-04-30T06:43:13.591ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67402025-04-30T06:43:13.591ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67412025-04-30T06:43:13.591ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67422025-04-30T06:43:13.591ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
67432025-04-30T06:43:13.592ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67442025-04-30T06:43:13.595ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67452025-04-30T06:43:13.603ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67462025-04-30T06:43:13.603ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67472025-04-30T06:43:13.603ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
67482025-04-30T06:43:13.604ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67492025-04-30T06:43:13.606ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75dd2634-80ac-4930-8ba9-065fb4002fa2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67502025-04-30T06:43:13.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
67512025-04-30T06:43:13.608ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67522025-04-30T06:43:13.608ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
67532025-04-30T06:43:13.609ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67542025-04-30T06:43:13.622ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8586d27f-94d0-4410-924b-3ca81bad5d47", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67552025-04-30T06:43:13.625ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
67562025-04-30T06:43:13.630ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3af4e0b-932d-4c77-9d7d-fe7c6f05c4ff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67572025-04-30T06:43:13.631ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
67582025-04-30T06:43:13.632ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0eaf556-8cf2-4c89-ad0b-63b92e6b9475", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67592025-04-30T06:43:13.632ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
67602025-04-30T06:43:13.635ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
67612025-04-30T06:43:13.636ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67622025-04-30T06:43:13.636ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 05d7b539-902e-478f-808e-3e833605cde2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67632025-04-30T06:43:13.637ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 05d7b539-902e-478f-808e-3e833605cde2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67642025-04-30T06:43:13.637ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 05d7b539-902e-478f-808e-3e833605cde2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67652025-04-30T06:43:13.638ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 05d7b539-902e-478f-808e-3e833605cde2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67662025-04-30T06:43:13.639ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "982fac6c-124f-4fc3-a8f9-46be32f4b3c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67672025-04-30T06:43:13.639ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
67682025-04-30T06:43:13.645ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 8093 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 05d7b539-902e-478f-808e-3e833605cde2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
67692025-04-30T06:43:13.646ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d7b539-902e-478f-808e-3e833605cde2", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67702025-04-30T06:43:13.647ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67712025-04-30T06:43:13.647ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37fc5370-0c1c-45c1-aa88-cf1076992678", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67722025-04-30T06:43:13.647ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
67732025-04-30T06:43:13.649ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67742025-04-30T06:43:13.650ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67752025-04-30T06:43:13.650ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67762025-04-30T06:43:13.652ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67772025-04-30T06:43:13.653ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a55c45b-69b5-49a4-af21-d971a225e626", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67782025-04-30T06:43:13.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
67792025-04-30T06:43:13.656ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fea9f21-871e-4c12-ab06-d2b3dae73a3e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67802025-04-30T06:43:13.656ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
67812025-04-30T06:43:13.657ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 8236 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
67822025-04-30T06:43:13.658ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bdbcdcc-6e4d-4cdd-b1a8-e3b512f2c926", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67832025-04-30T06:43:13.658ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67842025-04-30T06:43:13.658ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
67852025-04-30T06:43:13.658ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
67862025-04-30T06:43:13.661ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
67872025-04-30T06:43:13.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a5ace2-7121-44d3-9379-35b8f9785904", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67882025-04-30T06:43:13.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
67892025-04-30T06:43:13.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f49f80b3-58fd-4036-9e55-bf7132fd4eb6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67902025-04-30T06:43:13.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
67912025-04-30T06:43:13.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fc06dd-2975-4692-bbdb-aa72c05e83f9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67922025-04-30T06:43:13.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
67932025-04-30T06:43:13.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2593fc8d-5684-4e2e-ab63-4ad9dfc3521f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67942025-04-30T06:43:13.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
67952025-04-30T06:43:13.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5cea7ed-d4d0-4e64-9f2d-7bc0723c6c73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67962025-04-30T06:43:13.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
67972025-04-30T06:43:13.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1020f50f-142d-470e-9b22-5f60dc1267a9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
67982025-04-30T06:43:13.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
67992025-04-30T06:43:13.674ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee7a43e5-9220-4ba0-a2b8-8ec5f46e466f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
68002025-04-30T06:43:13.674ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
68012025-04-30T06:43:13.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72156ce-2b24-429c-abcd-fb0e3e1f4e00", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
68022025-04-30T06:43:13.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
68032025-04-30T06:43:13.676ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3a132d1-96d2-4442-aba3-087280e9309e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
68042025-04-30T06:43:13.676ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
68052025-04-30T06:43:13.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f13f233d-b18f-4437-8c13-d1b856981292", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
68062025-04-30T06:43:13.677ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68072025-04-30T06:43:13.677ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68082025-04-30T06:43:13.677ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68092025-04-30T06:43:13.677ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68102025-04-30T06:43:13.678ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68112025-04-30T06:43:13.681ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68122025-04-30T06:43:13.698ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68132025-04-30T06:43:13.698ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68142025-04-30T06:43:13.698ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68152025-04-30T06:43:13.698ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68162025-04-30T06:43:13.698ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68172025-04-30T06:43:13.698ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68182025-04-30T06:43:13.701ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68192025-04-30T06:43:13.706ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68202025-04-30T06:43:13.706ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68212025-04-30T06:43:13.706ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68222025-04-30T06:43:13.708ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68232025-04-30T06:43:13.708ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68242025-04-30T06:43:13.708ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68252025-04-30T06:43:13.711ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68262025-04-30T06:43:13.714ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68272025-04-30T06:43:13.714ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68282025-04-30T06:43:13.714ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68292025-04-30T06:43:13.789ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
68302025-04-30T06:43:13.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
68312025-04-30T06:43:13.790ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68322025-04-30T06:43:13.790ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68332025-04-30T06:43:13.790ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68342025-04-30T06:43:13.790ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
68352025-04-30T06:43:13.791ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68362025-04-30T06:43:13.791ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:46442 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68372025-04-30T06:43:13.791ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68382025-04-30T06:43:13.791ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68392025-04-30T06:43:13.792ZINFOtest_snapshot_unwind (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:49133 remote_addr = 127.0.0.1:60571
68402025-04-30T06:43:13.792ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:46442 req_id = 1b263c0f-87eb-471a-96a9-8409a5901adb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68412025-04-30T06:43:13.793ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 544 local_addr = [::1]:45714 method = GET remote_addr = [::1]:46442 req_id = 1b263c0f-87eb-471a-96a9-8409a5901adb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68422025-04-30T06:43:13.793ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = 14c0551e-1b04-457d-9504-408b838c666a uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
68432025-04-30T06:43:13.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b263c0f-87eb-471a-96a9-8409a5901adb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:12 GMT"} })
68442025-04-30T06:43:13.795ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68452025-04-30T06:43:13.795ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68462025-04-30T06:43:13.795ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
68472025-04-30T06:43:13.797ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68482025-04-30T06:43:13.797ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68492025-04-30T06:43:13.797ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68502025-04-30T06:43:13.798ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68512025-04-30T06:43:13.799ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68522025-04-30T06:43:13.809ZINFOtest_snapshot_unwind (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 15535 local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = 14c0551e-1b04-457d-9504-408b838c666a response_code = 200 uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
68532025-04-30T06:43:13.809ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68542025-04-30T06:43:13.815ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68552025-04-30T06:43:13.815ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68562025-04-30T06:43:13.815ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
68572025-04-30T06:43:13.815ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68582025-04-30T06:43:13.817ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68592025-04-30T06:43:13.832ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68602025-04-30T06:43:13.832ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68612025-04-30T06:43:13.832ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
68622025-04-30T06:43:13.832ZDEBGtest_snapshot_unwind (results-sink): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68632025-04-30T06:43:13.833ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68642025-04-30T06:43:13.833ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 18 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68652025-04-30T06:43:13.835ZDEBGtest_snapshot_unwind (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68662025-04-30T06:43:13.835ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
68672025-04-30T06:43:13.835ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68682025-04-30T06:43:13.839ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68692025-04-30T06:43:13.839ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68702025-04-30T06:43:13.839ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68712025-04-30T06:43:13.840ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68722025-04-30T06:43:13.842ZTRACtest_snapshot_unwind (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68732025-04-30T06:43:13.842ZDEBGtest_snapshot_unwind (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:13.835073091Z } timeseries_name = http_service:request_latency_histogram
68742025-04-30T06:43:13.850ZDEBGtest_snapshot_unwind (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68752025-04-30T06:43:13.850ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
68762025-04-30T06:43:13.854ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68772025-04-30T06:43:13.854ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
68782025-04-30T06:43:13.855ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68792025-04-30T06:43:13.856ZTRACtest_snapshot_unwind (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68802025-04-30T06:43:13.856ZDEBGtest_snapshot_unwind (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:13.850578514Z } timeseries_name = collection_target:cpus_provisioned
68812025-04-30T06:43:13.856ZDEBGtest_snapshot_unwind (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68822025-04-30T06:43:13.856ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
68832025-04-30T06:43:13.860ZTRACtest_snapshot_unwind (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68842025-04-30T06:43:13.860ZDEBGtest_snapshot_unwind (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:13.856845812Z } timeseries_name = collection_target:ram_provisioned
68852025-04-30T06:43:13.861ZDEBGtest_snapshot_unwind (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68862025-04-30T06:43:13.861ZTRACtest_snapshot_unwind (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
68872025-04-30T06:43:13.866ZTRACtest_snapshot_unwind (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68882025-04-30T06:43:13.866ZDEBGtest_snapshot_unwind (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:43:13.861043976Z } timeseries_name = collection_target:virtual_disk_space_provisioned
68892025-04-30T06:43:13.883ZDEBGtest_snapshot_unwind (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
68902025-04-30T06:43:13.884ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
68912025-04-30T06:43:13.894ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
68922025-04-30T06:43:13.901ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 12 table_name = fields_string
68932025-04-30T06:43:13.901ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
68942025-04-30T06:43:13.908ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = fields_u16
68952025-04-30T06:43:13.908ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
68962025-04-30T06:43:13.931ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 17 table_name = fields_uuid
68972025-04-30T06:43:13.931ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
68982025-04-30T06:43:13.941ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
68992025-04-30T06:43:13.945ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = measurements_histogramu64
69002025-04-30T06:43:13.945ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 13 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
69012025-04-30T06:43:13.951ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 13 table_name = measurements_i64
69022025-04-30T06:43:13.951ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69032025-04-30T06:43:13.952ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
69042025-04-30T06:43:13.952ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
69052025-04-30T06:43:13.952ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 518.970675ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
69062025-04-30T06:43:13.953ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
69072025-04-30T06:43:13.953ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
69082025-04-30T06:43:13.953ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69092025-04-30T06:43:13.953ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69102025-04-30T06:43:13.953ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69112025-04-30T06:43:13.954ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69122025-04-30T06:43:13.955ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
69132025-04-30T06:43:13.958ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69142025-04-30T06:43:13.973ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
69152025-04-30T06:43:13.973ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69162025-04-30T06:43:13.973ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69172025-04-30T06:43:13.974ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69182025-04-30T06:43:13.976ZTRACtest_snapshot_unwind (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69192025-04-30T06:43:13.976ZTRACtest_snapshot_unwind (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69202025-04-30T06:43:13.987ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
69212025-04-30T06:43:13.987ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
69222025-04-30T06:43:13.989ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 72 ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 r............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50495 req_id = 5a05342c-c5e3-4f06-8e3f-22590a1f157c
69232025-04-30T06:43:13.989ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29355,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50495 req_id = 5a05342c-c5e3-4f06-8e3f-22590a1f157c
69242025-04-30T06:43:13.990ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
69252025-04-30T06:43:13.990ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69262025-04-30T06:43:13.990ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50495 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5a05342c-c5e3-4f06-8e3f-22590a1f157c
69272025-04-30T06:43:13.993ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4258, tv_nsec: 792239202 } })
69282025-04-30T06:43:13.993ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
69292025-04-30T06:43:13.994ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f7 76 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .v..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62340 req_id = 4b3c5634-90f2-4980-9733-450d8781cd31
69302025-04-30T06:43:13.994ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63350,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62340 req_id = 4b3c5634-90f2-4980-9733-450d8781cd31
69312025-04-30T06:43:13.995ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69322025-04-30T06:43:13.995ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62340 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4b3c5634-90f2-4980-9733-450d8781cd31
69332025-04-30T06:43:14.057ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
69342025-04-30T06:43:14.058ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69352025-04-30T06:43:14.058ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69362025-04-30T06:43:14.058ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69372025-04-30T06:43:14.058ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
69382025-04-30T06:43:14.060ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69392025-04-30T06:43:14.061ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ad 63 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .c..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62581 req_id = dedb10cf-b38d-4afe-9203-6c7d92280174
69402025-04-30T06:43:14.061ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44387,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62581 req_id = dedb10cf-b38d-4afe-9203-6c7d92280174
69412025-04-30T06:43:14.061ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
69422025-04-30T06:43:14.061ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44387, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62581 req_id = dedb10cf-b38d-4afe-9203-6c7d92280174
69432025-04-30T06:43:14.061ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
69442025-04-30T06:43:14.061ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
69452025-04-30T06:43:14.061ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
69462025-04-30T06:43:14.063ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69472025-04-30T06:43:14.068ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69482025-04-30T06:43:14.068ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69492025-04-30T06:43:14.074ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
69502025-04-30T06:43:14.075ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69512025-04-30T06:43:14.076ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dca3c9b7-61ae-4e57-a63b-391335bed946", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69522025-04-30T06:43:14.079ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
69532025-04-30T06:43:14.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b361f65-e34c-4628-a006-196665750c68", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69542025-04-30T06:43:14.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
69552025-04-30T06:43:14.081ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69562025-04-30T06:43:14.081ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
69572025-04-30T06:43:14.081ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69582025-04-30T06:43:14.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80ae3112-7874-43a7-bf5f-4d7b9444298e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69592025-04-30T06:43:14.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
69602025-04-30T06:43:14.098ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b80d992-cb71-461b-bbfa-073ea984617f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69612025-04-30T06:43:14.098ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
69622025-04-30T06:43:14.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "470b1ad3-0600-4529-a43e-ada16623c2ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69632025-04-30T06:43:14.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
69642025-04-30T06:43:14.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98d5472f-781b-4796-bc24-46d561beee40", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69652025-04-30T06:43:14.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
69662025-04-30T06:43:14.119ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
69672025-04-30T06:43:14.120ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac5c3c8b-2be7-46fd-bad1-c6c3aef936af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69682025-04-30T06:43:14.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
69692025-04-30T06:43:14.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2e69e0b-6734-4dff-8b27-9cc3146f62b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69702025-04-30T06:43:14.125ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
69712025-04-30T06:43:14.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c28ebec8-0c6e-4405-8f7d-fb3311877092", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69722025-04-30T06:43:14.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
69732025-04-30T06:43:14.128ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90098ced-d686-4afa-8e3f-b157b7d0c584", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69742025-04-30T06:43:14.128ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
69752025-04-30T06:43:14.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e9d6240-2423-4b7b-9e8c-27cd998621dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69762025-04-30T06:43:14.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
69772025-04-30T06:43:14.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e771999a-0748-4297-9a2e-26a8dd2ee5ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69782025-04-30T06:43:14.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
69792025-04-30T06:43:14.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c3edd6-25ab-47e5-9d7c-5d163b36487f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69802025-04-30T06:43:14.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
69812025-04-30T06:43:14.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e13af555-f276-4d04-a1c8-c9ab20df0886", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69822025-04-30T06:43:14.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
69832025-04-30T06:43:14.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d92aa83c-41d5-49e3-b94d-c8bc4bb504cc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69842025-04-30T06:43:14.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
69852025-04-30T06:43:14.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e11a08f-9487-4930-af65-342c31aa6107", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69862025-04-30T06:43:14.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
69872025-04-30T06:43:14.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67cd9419-1e76-44b4-946d-3c6800cf850c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69882025-04-30T06:43:14.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
69892025-04-30T06:43:14.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81aeadde-878a-4e94-b3f9-c11f56b77baf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69902025-04-30T06:43:14.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
69912025-04-30T06:43:14.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48bdeaf7-ccb7-4bbb-b3ef-fd04cbbf36a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69922025-04-30T06:43:14.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
69932025-04-30T06:43:14.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8634729f-0807-4e4d-9dab-6dd714008f41", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69942025-04-30T06:43:14.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
69952025-04-30T06:43:14.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98fa893f-44cd-4c8c-bd1e-0a27e41ad7d3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69962025-04-30T06:43:14.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
69972025-04-30T06:43:14.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03c40857-6f26-46fc-a49a-64c8e1cbc50b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
69982025-04-30T06:43:14.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
69992025-04-30T06:43:14.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "105d9049-b8c3-49e2-b10e-2193c8e451d6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70002025-04-30T06:43:14.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
70012025-04-30T06:43:14.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c58482f-f835-4895-ba1c-f49fb2a7a484", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70022025-04-30T06:43:14.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
70032025-04-30T06:43:14.169ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70042025-04-30T06:43:14.169ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70052025-04-30T06:43:14.170ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70062025-04-30T06:43:14.170ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
70072025-04-30T06:43:14.170ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70082025-04-30T06:43:14.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1d45b66-3206-4743-bd4a-ccf310288982", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70092025-04-30T06:43:14.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
70102025-04-30T06:43:14.173ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70112025-04-30T06:43:14.179ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70122025-04-30T06:43:14.179ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70132025-04-30T06:43:14.179ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
70142025-04-30T06:43:14.179ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70152025-04-30T06:43:14.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbd2a3f-fc97-4339-9a2c-ff7ae7de09f2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70162025-04-30T06:43:14.181ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
70172025-04-30T06:43:14.182ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70182025-04-30T06:43:14.182ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
70192025-04-30T06:43:14.182ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70202025-04-30T06:43:14.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6de7e3d2-2682-4de7-ad61-899977819f8d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70212025-04-30T06:43:14.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
70222025-04-30T06:43:14.187ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34023794-b3cb-4422-9fd7-80ae7ccc7120", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70232025-04-30T06:43:14.188ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
70242025-04-30T06:43:14.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d32a8e6b-eb11-41e2-9c22-7023e1d41aa3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70252025-04-30T06:43:14.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
70262025-04-30T06:43:14.193ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6126483-74f4-4a74-b456-ab461fd1ecef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70272025-04-30T06:43:14.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
70282025-04-30T06:43:14.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cff30b4-94a7-42c9-aed8-f22d22afa3d1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70292025-04-30T06:43:14.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
70302025-04-30T06:43:14.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d525fc6-5593-4f1a-a507-10861103e3d7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70312025-04-30T06:43:14.199ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
70322025-04-30T06:43:14.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de90dc38-5435-4136-adc2-2e0bb79cdd3e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70332025-04-30T06:43:14.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
70342025-04-30T06:43:14.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa3d66a5-972d-4f92-986e-e730941ab131", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70352025-04-30T06:43:14.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
70362025-04-30T06:43:14.202ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d71b7d-0899-4d83-bf73-ea9f8b6dd440", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70372025-04-30T06:43:14.203ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
70382025-04-30T06:43:14.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee412316-5295-4882-9627-a7702712b5a8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70392025-04-30T06:43:14.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
70402025-04-30T06:43:14.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46b58613-e4a3-4356-9b59-930cdf69ca7a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70412025-04-30T06:43:14.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
70422025-04-30T06:43:14.213ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
70432025-04-30T06:43:14.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44141ad0-a3d9-4afc-bef5-e549b2a7ef8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70442025-04-30T06:43:14.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
70452025-04-30T06:43:14.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8e1b619-22b7-42f7-89c7-791b8b61d6f7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70462025-04-30T06:43:14.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
70472025-04-30T06:43:14.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a7eefe-806c-4f59-a8ea-d6cfe3befa4f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70482025-04-30T06:43:14.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
70492025-04-30T06:43:14.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e3e98a-78ed-48ac-a177-d94d8d392233", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70502025-04-30T06:43:14.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
70512025-04-30T06:43:14.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93cb903b-425b-41df-8f4a-27dc7c26d37c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70522025-04-30T06:43:14.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
70532025-04-30T06:43:14.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "108fba6e-4e2e-4f19-834a-e5b177425418", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70542025-04-30T06:43:14.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
70552025-04-30T06:43:14.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "228e7e61-7a08-4369-b3ca-190f82f7e8a4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70562025-04-30T06:43:14.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
70572025-04-30T06:43:14.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d871898b-91c1-42d3-99ba-e50405fcf61e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70582025-04-30T06:43:14.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70592025-04-30T06:43:14.231ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70602025-04-30T06:43:14.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70612025-04-30T06:43:14.231ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70622025-04-30T06:43:14.232ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70632025-04-30T06:43:14.234ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70642025-04-30T06:43:14.238ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70652025-04-30T06:43:14.238ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70662025-04-30T06:43:14.238ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70672025-04-30T06:43:14.238ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70682025-04-30T06:43:14.238ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70692025-04-30T06:43:14.239ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70702025-04-30T06:43:14.242ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70712025-04-30T06:43:14.247ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70722025-04-30T06:43:14.247ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70732025-04-30T06:43:14.247ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70742025-04-30T06:43:14.248ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70752025-04-30T06:43:14.318ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
70762025-04-30T06:43:14.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
70772025-04-30T06:43:14.319ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70782025-04-30T06:43:14.319ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70792025-04-30T06:43:14.319ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70802025-04-30T06:43:14.319ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
70812025-04-30T06:43:14.319ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70822025-04-30T06:43:14.320ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:41796 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70832025-04-30T06:43:14.321ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:41796 req_id = 375a1ec6-1f7c-4095-a2e4-b33ea4d7a80f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70842025-04-30T06:43:14.326ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 5334 local_addr = [::1]:45714 method = GET remote_addr = [::1]:41796 req_id = 375a1ec6-1f7c-4095-a2e4-b33ea4d7a80f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70852025-04-30T06:43:14.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "375a1ec6-1f7c-4095-a2e4-b33ea4d7a80f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
70862025-04-30T06:43:14.330ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70872025-04-30T06:43:14.330ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70882025-04-30T06:43:14.330ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
70892025-04-30T06:43:14.332ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70902025-04-30T06:43:14.332ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70912025-04-30T06:43:14.333ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70922025-04-30T06:43:14.333ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70932025-04-30T06:43:14.333ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70942025-04-30T06:43:14.333ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70952025-04-30T06:43:14.341ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70962025-04-30T06:43:14.341ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70972025-04-30T06:43:14.341ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
70982025-04-30T06:43:14.341ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70992025-04-30T06:43:14.345ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71002025-04-30T06:43:14.351ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71012025-04-30T06:43:14.351ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71022025-04-30T06:43:14.351ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71032025-04-30T06:43:14.352ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71042025-04-30T06:43:14.356ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71052025-04-30T06:43:14.356ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71062025-04-30T06:43:14.357ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71072025-04-30T06:43:14.411ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
71082025-04-30T06:43:14.412ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
71092025-04-30T06:43:14.412ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 459.118203ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
71102025-04-30T06:43:14.412ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
71112025-04-30T06:43:14.412ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
71122025-04-30T06:43:14.412ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71132025-04-30T06:43:14.412ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71142025-04-30T06:43:14.412ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71152025-04-30T06:43:14.413ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71162025-04-30T06:43:14.415ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71172025-04-30T06:43:14.420ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71182025-04-30T06:43:14.420ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71192025-04-30T06:43:14.420ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71202025-04-30T06:43:14.421ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71212025-04-30T06:43:14.426ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
71222025-04-30T06:43:14.427ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
71232025-04-30T06:43:14.427ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
71242025-04-30T06:43:14.428ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: da e5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55998 req_id = 9bd98cf0-0b56-44e9-9967-e43b5b4f096b
71252025-04-30T06:43:14.428ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55998 req_id = 9bd98cf0-0b56-44e9-9967-e43b5b4f096b
71262025-04-30T06:43:14.428ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
71272025-04-30T06:43:14.428ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71282025-04-30T06:43:14.431ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55998 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9bd98cf0-0b56-44e9-9967-e43b5b4f096b
71292025-04-30T06:43:14.431ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 230339138 } })
71302025-04-30T06:43:14.431ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
71312025-04-30T06:43:14.432ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0a f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57124 req_id = 6303ea95-4b83-40db-ab03-1d7a9dcc763a
71322025-04-30T06:43:14.432ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2809,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57124 req_id = 6303ea95-4b83-40db-ab03-1d7a9dcc763a
71332025-04-30T06:43:14.432ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71342025-04-30T06:43:14.433ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57124 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6303ea95-4b83-40db-ab03-1d7a9dcc763a
71352025-04-30T06:43:14.507ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
71362025-04-30T06:43:14.508ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71372025-04-30T06:43:14.508ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71382025-04-30T06:43:14.508ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71392025-04-30T06:43:14.508ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71402025-04-30T06:43:14.509ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71412025-04-30T06:43:14.509ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b6 45 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .E..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52527 req_id = 46582208-5d78-4551-93e6-3f6c4b41c6af
71422025-04-30T06:43:14.510ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46661,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52527 req_id = 46582208-5d78-4551-93e6-3f6c4b41c6af
71432025-04-30T06:43:14.510ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
71442025-04-30T06:43:14.510ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46661, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52527 req_id = 46582208-5d78-4551-93e6-3f6c4b41c6af
71452025-04-30T06:43:14.510ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
71462025-04-30T06:43:14.511ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
71472025-04-30T06:43:14.511ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
71482025-04-30T06:43:14.511ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71492025-04-30T06:43:14.519ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71502025-04-30T06:43:14.519ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71512025-04-30T06:43:14.519ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71522025-04-30T06:43:14.520ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71532025-04-30T06:43:14.523ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "37649b98-2440-4f4b-aa02-7e307c0224b0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71542025-04-30T06:43:14.530ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
71552025-04-30T06:43:14.530ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71562025-04-30T06:43:14.532ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71572025-04-30T06:43:14.533ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71582025-04-30T06:43:14.539ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f566de-d1b2-4da9-8b94-4a42baf90815", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71592025-04-30T06:43:14.539ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
71602025-04-30T06:43:14.544ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2938a45e-8fe1-4da2-a737-2e2fb36f3e28", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71612025-04-30T06:43:14.544ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
71622025-04-30T06:43:14.550ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1237b2f7-cd85-40a7-957c-58505c1bbc51", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71632025-04-30T06:43:14.550ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
71642025-04-30T06:43:14.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d84ed940-b3da-401c-a843-5ef1e05b7575", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71652025-04-30T06:43:14.559ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
71662025-04-30T06:43:14.563ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c41f9b-f75c-4ee7-9eac-a8422c1e8aec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71672025-04-30T06:43:14.563ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
71682025-04-30T06:43:14.567ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
71692025-04-30T06:43:14.576ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df8ad225-e062-43a8-924e-7643ef91c2f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71702025-04-30T06:43:14.583ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
71712025-04-30T06:43:14.588ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f373304-40d8-4731-a229-ed76fbb37f26", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71722025-04-30T06:43:14.588ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
71732025-04-30T06:43:14.591ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa8e8e6-0529-4daf-a2eb-1a2717ead859", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71742025-04-30T06:43:14.591ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
71752025-04-30T06:43:14.592ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "faefa94d-c782-457a-8433-3f43d5431aca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71762025-04-30T06:43:14.593ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
71772025-04-30T06:43:14.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9158fc00-f1eb-4346-b045-7e90c3380d4a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71782025-04-30T06:43:14.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
71792025-04-30T06:43:14.597ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e04d8b52-84dc-4d25-9bb2-14b5e96adb68", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71802025-04-30T06:43:14.599ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
71812025-04-30T06:43:14.604ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90733b04-c028-4d48-bf33-85e0725030e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71822025-04-30T06:43:14.605ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
71832025-04-30T06:43:14.611ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2993d9a0-323f-4794-9c02-b33efc95a5b2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71842025-04-30T06:43:14.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
71852025-04-30T06:43:14.620ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71862025-04-30T06:43:14.620ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71872025-04-30T06:43:14.620ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71882025-04-30T06:43:14.620ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71892025-04-30T06:43:14.622ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71902025-04-30T06:43:14.622ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27bad7fd-d08c-4ca1-aa11-396a5ecd3f62", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71912025-04-30T06:43:14.623ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
71922025-04-30T06:43:14.627ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49ee7528-591b-4120-8f7e-7f4c4f0e6d64", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
71932025-04-30T06:43:14.627ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
71942025-04-30T06:43:14.627ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71952025-04-30T06:43:14.633ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71962025-04-30T06:43:14.633ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71972025-04-30T06:43:14.633ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
71982025-04-30T06:43:14.634ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71992025-04-30T06:43:14.636ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72002025-04-30T06:43:14.636ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
72012025-04-30T06:43:14.637ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72022025-04-30T06:43:14.639ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abd6aad8-4127-4d3f-b511-c9102a2369cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72032025-04-30T06:43:14.639ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
72042025-04-30T06:43:14.643ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f4b4068-b9a3-4cac-aa44-3bf8863027b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72052025-04-30T06:43:14.644ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
72062025-04-30T06:43:14.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9200017-f1ba-4812-b0c5-511c24bacc18", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72072025-04-30T06:43:14.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
72082025-04-30T06:43:14.648ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce73a35b-09da-4e0b-bb53-11ca7803cb35", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72092025-04-30T06:43:14.648ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
72102025-04-30T06:43:14.650ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2275358-bdbd-4e27-a65f-94a9cc29d0b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72112025-04-30T06:43:14.650ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
72122025-04-30T06:43:14.651ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "815e61fb-53b2-4f58-b61d-1c183d1b23e5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72132025-04-30T06:43:14.651ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
72142025-04-30T06:43:14.653ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a5dd7eb-7dae-4f74-89fa-22163ff039ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72152025-04-30T06:43:14.653ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
72162025-04-30T06:43:14.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "026f0919-78a4-4699-bcea-43dec5731710", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72172025-04-30T06:43:14.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
72182025-04-30T06:43:14.659ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab2860a-859c-4bfb-b8a9-7871abb47135", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72192025-04-30T06:43:14.659ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
72202025-04-30T06:43:14.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d714538-6130-46b6-a526-b5e51fc0321f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72212025-04-30T06:43:14.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
72222025-04-30T06:43:14.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9403cbe-da60-4922-b7d0-1c597dcda98f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72232025-04-30T06:43:14.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
72242025-04-30T06:43:14.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ce3edc-f61a-40d3-b5e8-1789838f34f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72252025-04-30T06:43:14.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
72262025-04-30T06:43:14.669ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
72272025-04-30T06:43:14.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66488ed1-35d6-4636-b61b-061d18a81dad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72282025-04-30T06:43:14.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
72292025-04-30T06:43:14.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d154f3f3-46e8-4be3-a041-15ff6ceae49d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72302025-04-30T06:43:14.674ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
72312025-04-30T06:43:14.674ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9e266b-2fdf-4bb7-90b4-d599e7ffe466", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72322025-04-30T06:43:14.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
72332025-04-30T06:43:14.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8fc4faf-961d-4782-8c3b-48e22a593d21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72342025-04-30T06:43:14.676ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
72352025-04-30T06:43:14.678ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8fa139a-5cfb-4d34-899e-712323201315", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72362025-04-30T06:43:14.678ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
72372025-04-30T06:43:14.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb52afc3-2935-4cd2-a406-30c74c46b35f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72382025-04-30T06:43:14.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
72392025-04-30T06:43:14.681ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3acedf7-73f0-4fec-a8cc-64b6e05d4b92", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72402025-04-30T06:43:14.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
72412025-04-30T06:43:14.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bfa9f36-b29b-489d-86d3-0ed92cb6bf21", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72422025-04-30T06:43:14.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
72432025-04-30T06:43:14.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec027a0-8baa-416d-9c91-45756c092b9a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72442025-04-30T06:43:14.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
72452025-04-30T06:43:14.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21373292-0816-4a6d-bbd9-13aa701f5ab5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72462025-04-30T06:43:14.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
72472025-04-30T06:43:14.696ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80c6bde4-c1ed-44b0-93ec-4afd1a1d42ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72482025-04-30T06:43:14.696ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
72492025-04-30T06:43:14.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f06e2c90-e312-4308-be7f-2109a759a172", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72502025-04-30T06:43:14.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
72512025-04-30T06:43:14.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72522025-04-30T06:43:14.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72532025-04-30T06:43:14.706ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72542025-04-30T06:43:14.708ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72552025-04-30T06:43:14.708ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72562025-04-30T06:43:14.708ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72572025-04-30T06:43:14.710ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67f3384d-e143-4a06-a7f5-605d258ac59f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72582025-04-30T06:43:14.711ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
72592025-04-30T06:43:14.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78714e59-580e-4fb6-a0d9-24f35265d8b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72602025-04-30T06:43:14.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
72612025-04-30T06:43:14.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3150a2c4-0abc-402a-80e9-4a5ed1227720", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72622025-04-30T06:43:14.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
72632025-04-30T06:43:14.714ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4ef5daf-cf50-4052-aef9-076010ac52fb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72642025-04-30T06:43:14.715ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
72652025-04-30T06:43:14.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d3350a7-4d9c-46fe-910c-38edaa89faf5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
72662025-04-30T06:43:14.716ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72672025-04-30T06:43:14.717ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72682025-04-30T06:43:14.717ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72692025-04-30T06:43:14.717ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72702025-04-30T06:43:14.717ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72712025-04-30T06:43:14.720ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72722025-04-30T06:43:14.723ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72732025-04-30T06:43:14.723ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72742025-04-30T06:43:14.724ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72752025-04-30T06:43:14.724ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72762025-04-30T06:43:14.724ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72772025-04-30T06:43:14.724ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72782025-04-30T06:43:14.725ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72792025-04-30T06:43:14.725ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72802025-04-30T06:43:14.726ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72812025-04-30T06:43:14.726ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
72822025-04-30T06:43:14.726ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72832025-04-30T06:43:14.728ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72842025-04-30T06:43:14.731ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72852025-04-30T06:43:14.732ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72862025-04-30T06:43:14.732ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72872025-04-30T06:43:14.732ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72882025-04-30T06:43:14.733ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72892025-04-30T06:43:14.739ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72902025-04-30T06:43:14.739ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72912025-04-30T06:43:14.740ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
72922025-04-30T06:43:14.740ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72932025-04-30T06:43:14.743ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72942025-04-30T06:43:14.743ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
72952025-04-30T06:43:14.745ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72962025-04-30T06:43:14.799ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
72972025-04-30T06:43:14.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
72982025-04-30T06:43:14.800ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40721 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72992025-04-30T06:43:14.800ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40721 req_id = 5a300d1e-3ae0-4fbc-966f-bf0232c3cf39 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73002025-04-30T06:43:14.801ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 492 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40721 req_id = 5a300d1e-3ae0-4fbc-966f-bf0232c3cf39 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73012025-04-30T06:43:14.801ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a300d1e-3ae0-4fbc-966f-bf0232c3cf39", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73022025-04-30T06:43:14.805ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73032025-04-30T06:43:14.806ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73042025-04-30T06:43:14.806ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
73052025-04-30T06:43:14.811ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73062025-04-30T06:43:14.811ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73072025-04-30T06:43:14.811ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73082025-04-30T06:43:14.811ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73092025-04-30T06:43:14.813ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73102025-04-30T06:43:14.817ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73112025-04-30T06:43:14.822ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73122025-04-30T06:43:14.823ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73132025-04-30T06:43:14.823ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73142025-04-30T06:43:14.824ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73152025-04-30T06:43:14.863ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
73162025-04-30T06:43:14.865ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73172025-04-30T06:43:14.865ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73182025-04-30T06:43:14.865ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 453.711069ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73192025-04-30T06:43:14.866ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
73202025-04-30T06:43:14.866ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73212025-04-30T06:43:14.866ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73222025-04-30T06:43:14.866ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73232025-04-30T06:43:14.866ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73242025-04-30T06:43:14.870ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73252025-04-30T06:43:14.874ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73262025-04-30T06:43:14.878ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73272025-04-30T06:43:14.878ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73282025-04-30T06:43:14.878ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73292025-04-30T06:43:14.879ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73302025-04-30T06:43:14.885ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
73312025-04-30T06:43:14.886ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73322025-04-30T06:43:14.887ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 55 8a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 U............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50482 req_id = fb983e9f-ddd3-4f53-96fc-325092403eba
73332025-04-30T06:43:14.887ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21898,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50482 req_id = fb983e9f-ddd3-4f53-96fc-325092403eba
73342025-04-30T06:43:14.887ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
73352025-04-30T06:43:14.887ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73362025-04-30T06:43:14.887ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50482 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fb983e9f-ddd3-4f53-96fc-325092403eba
73372025-04-30T06:43:14.888ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4259, tv_nsec: 686927721 } })
73382025-04-30T06:43:14.888ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
73392025-04-30T06:43:14.888ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e f7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63534 req_id = 389c4b42-7b02-4727-bec4-f9b2c51f0e23
73402025-04-30T06:43:14.888ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40695,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63534 req_id = 389c4b42-7b02-4727-bec4-f9b2c51f0e23
73412025-04-30T06:43:14.889ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73422025-04-30T06:43:14.889ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63534 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 389c4b42-7b02-4727-bec4-f9b2c51f0e23
73432025-04-30T06:43:14.968ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
73442025-04-30T06:43:14.970ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73452025-04-30T06:43:14.970ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73462025-04-30T06:43:14.970ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73472025-04-30T06:43:14.970ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
73482025-04-30T06:43:14.971ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73492025-04-30T06:43:14.971ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ef 1e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62011 req_id = 64df31eb-cd38-4a03-8020-1d01fdb414d8
73502025-04-30T06:43:14.973ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61214,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62011 req_id = 64df31eb-cd38-4a03-8020-1d01fdb414d8
73512025-04-30T06:43:14.973ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
73522025-04-30T06:43:14.973ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61214, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62011 req_id = 64df31eb-cd38-4a03-8020-1d01fdb414d8
73532025-04-30T06:43:14.974ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
73542025-04-30T06:43:14.974ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
73552025-04-30T06:43:14.974ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
73562025-04-30T06:43:14.975ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73572025-04-30T06:43:14.986ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73582025-04-30T06:43:14.986ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73592025-04-30T06:43:14.986ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
73602025-04-30T06:43:14.987ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73612025-04-30T06:43:14.987ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73622025-04-30T06:43:14.987ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73632025-04-30T06:43:14.988ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
73642025-04-30T06:43:14.989ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
73652025-04-30T06:43:14.997ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73662025-04-30T06:43:14.997ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
73672025-04-30T06:43:14.998ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73682025-04-30T06:43:14.999ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
73692025-04-30T06:43:14.999ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
73702025-04-30T06:43:15.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef1fbd1-cc49-4a78-8256-13cbea99dc11", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73712025-04-30T06:43:15.006ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
73722025-04-30T06:43:15.007ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
73732025-04-30T06:43:15.007ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73742025-04-30T06:43:15.009ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc470804-a46c-457b-ad28-432d57037052", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73752025-04-30T06:43:15.010ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
73762025-04-30T06:43:15.013ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2e2bd86-48f0-4f49-82b3-c34ccf077312", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73772025-04-30T06:43:15.014ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
73782025-04-30T06:43:15.044ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bd12b8e-f0e0-45fc-8eb6-d0a02b0a228e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73792025-04-30T06:43:15.044ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
73802025-04-30T06:43:15.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66bf313a-dde7-418e-8e75-732f329697be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73812025-04-30T06:43:15.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
73822025-04-30T06:43:15.050ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb5a59d6-5a5e-4bbf-a80a-d48bf369806d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73832025-04-30T06:43:15.050ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
73842025-04-30T06:43:15.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7895df44-648f-4ba4-835e-f3654fd67228", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73852025-04-30T06:43:15.056ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
73862025-04-30T06:43:15.061ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "596d879a-9ca8-44de-8125-a424a45f6d9c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:14 GMT"} })
73872025-04-30T06:43:15.061ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
73882025-04-30T06:43:15.066ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
73892025-04-30T06:43:15.066ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a09bf122-f5ed-4185-8bea-b125bd528ba8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
73902025-04-30T06:43:15.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
73912025-04-30T06:43:15.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92cdc9f7-c3f8-466d-9e3e-1c9cb06c09df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
73922025-04-30T06:43:15.074ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
73932025-04-30T06:43:15.078ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b0c77a-9fd3-44c7-b862-4897db70bb44", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
73942025-04-30T06:43:15.078ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
73952025-04-30T06:43:15.079ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f55cfc3-831b-4cab-8d18-7671f6f87c86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
73962025-04-30T06:43:15.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
73972025-04-30T06:43:15.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e94cb451-dcca-46a4-bf3b-7c28e2349b8c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
73982025-04-30T06:43:15.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
73992025-04-30T06:43:15.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "699389b1-b36f-4494-944d-06b7d7141944", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74002025-04-30T06:43:15.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
74012025-04-30T06:43:15.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1862c4f0-4987-427f-956a-a4e6b38f6131", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74022025-04-30T06:43:15.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
74032025-04-30T06:43:15.090ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74042025-04-30T06:43:15.116ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74052025-04-30T06:43:15.119ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74062025-04-30T06:43:15.122ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74072025-04-30T06:43:15.122ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74082025-04-30T06:43:15.127ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74092025-04-30T06:43:15.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d00e57b-8e1d-44ba-8d78-10d0e41fb74f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74102025-04-30T06:43:15.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
74112025-04-30T06:43:15.143ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74122025-04-30T06:43:15.151ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74132025-04-30T06:43:15.151ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74142025-04-30T06:43:15.151ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74152025-04-30T06:43:15.152ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74162025-04-30T06:43:15.155ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74172025-04-30T06:43:15.155ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74182025-04-30T06:43:15.155ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74192025-04-30T06:43:15.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "850eae07-c713-465e-93d6-1b932b674568", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74202025-04-30T06:43:15.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
74212025-04-30T06:43:15.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c50bd3ed-eb4d-4c59-8cd6-51b697abce20", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74222025-04-30T06:43:15.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
74232025-04-30T06:43:15.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "317dd681-eb2b-4d53-a3af-e1083b7c8d7d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74242025-04-30T06:43:15.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
74252025-04-30T06:43:15.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6f3bd43-4da7-433c-8011-e6d3d07ad38b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74262025-04-30T06:43:15.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
74272025-04-30T06:43:15.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f7b92a6-7917-4096-9446-8122f189a57c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74282025-04-30T06:43:15.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
74292025-04-30T06:43:15.196ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5747df4e-fedd-4edd-97bc-7ceca123663a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74302025-04-30T06:43:15.196ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
74312025-04-30T06:43:15.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d2b7b0-3532-41aa-94a4-247c31e75605", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74322025-04-30T06:43:15.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
74332025-04-30T06:43:15.202ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
74342025-04-30T06:43:15.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93880cfc-2010-4a74-a516-c77501e12132", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74352025-04-30T06:43:15.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
74362025-04-30T06:43:15.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a59c4f87-0acc-468f-badb-9a53ad61667d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74372025-04-30T06:43:15.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
74382025-04-30T06:43:15.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4402e83-f768-4855-87b8-f0b38261cd87", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74392025-04-30T06:43:15.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
74402025-04-30T06:43:15.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b3858c7-b309-4d35-a639-b1a5925cc82a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74412025-04-30T06:43:15.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
74422025-04-30T06:43:15.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "602486b3-acf8-4853-bbe2-4c858cfaab16", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74432025-04-30T06:43:15.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
74442025-04-30T06:43:15.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fdbcc3-e9ce-4a51-abb2-8df53839e9da", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74452025-04-30T06:43:15.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
74462025-04-30T06:43:15.237ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c112f77-fec2-46b4-94e5-3182660b05d7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74472025-04-30T06:43:15.237ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
74482025-04-30T06:43:15.238ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02b70381-f68d-4e4b-a648-67f660f7acdc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74492025-04-30T06:43:15.238ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
74502025-04-30T06:43:15.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79051edc-e151-49ce-824a-3767d349c613", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74512025-04-30T06:43:15.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
74522025-04-30T06:43:15.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98fea7a5-abf0-429b-bce4-5e04d6ddbb83", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74532025-04-30T06:43:15.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
74542025-04-30T06:43:15.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10647d4e-9cfd-4563-995e-5b44b1e27066", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74552025-04-30T06:43:15.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
74562025-04-30T06:43:15.243ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c7fbc52-aeeb-4e16-8644-d74faa14bb7f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74572025-04-30T06:43:15.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
74582025-04-30T06:43:15.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b2a9ac4-a718-47eb-ba80-0ca67fba9844", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74592025-04-30T06:43:15.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
74602025-04-30T06:43:15.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5f15da-53e3-41dc-a0f4-f6f3689ae2a3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74612025-04-30T06:43:15.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
74622025-04-30T06:43:15.255ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74632025-04-30T06:43:15.255ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74642025-04-30T06:43:15.255ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74652025-04-30T06:43:15.255ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74662025-04-30T06:43:15.257ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74672025-04-30T06:43:15.268ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74682025-04-30T06:43:15.273ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74692025-04-30T06:43:15.274ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74702025-04-30T06:43:15.274ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74712025-04-30T06:43:15.275ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74722025-04-30T06:43:15.280ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74732025-04-30T06:43:15.280ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
74742025-04-30T06:43:15.281ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74752025-04-30T06:43:15.291ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af64b45-a260-4ddc-86b4-bcf31ff3b2a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74762025-04-30T06:43:15.291ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
74772025-04-30T06:43:15.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78194fd1-69be-44cf-a5c5-2ba371704705", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74782025-04-30T06:43:15.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
74792025-04-30T06:43:15.309ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f7cc39e-fb52-4e75-9d7e-8498e85b7cee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74802025-04-30T06:43:15.310ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
74812025-04-30T06:43:15.312ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa0db37-c10a-40e3-9c40-2a5deb7de20e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74822025-04-30T06:43:15.312ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
74832025-04-30T06:43:15.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc77ab10-9a1f-4fa7-ba1f-92daf90379c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74842025-04-30T06:43:15.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
74852025-04-30T06:43:15.317ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54db7ce4-5feb-41c2-9296-2dde0b61dc2b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74862025-04-30T06:43:15.317ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
74872025-04-30T06:43:15.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f118fef-888e-4291-93b8-b37e7db41eed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74882025-04-30T06:43:15.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
74892025-04-30T06:43:15.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40989320-d10d-48b9-98f8-0dc3c984a350", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
74902025-04-30T06:43:15.323ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74912025-04-30T06:43:15.323ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74922025-04-30T06:43:15.323ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74932025-04-30T06:43:15.323ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74942025-04-30T06:43:15.324ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74952025-04-30T06:43:15.324ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
74962025-04-30T06:43:15.326ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74972025-04-30T06:43:15.332ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74982025-04-30T06:43:15.332ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74992025-04-30T06:43:15.332ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75002025-04-30T06:43:15.332ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75012025-04-30T06:43:15.332ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75022025-04-30T06:43:15.333ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75032025-04-30T06:43:15.335ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75042025-04-30T06:43:15.340ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75052025-04-30T06:43:15.342ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75062025-04-30T06:43:15.343ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75072025-04-30T06:43:15.344ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75082025-04-30T06:43:15.413ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
75092025-04-30T06:43:15.414ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
75102025-04-30T06:43:15.414ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75112025-04-30T06:43:15.415ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75122025-04-30T06:43:15.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75132025-04-30T06:43:15.415ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
75142025-04-30T06:43:15.417ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75152025-04-30T06:43:15.417ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40464 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
75162025-04-30T06:43:15.419ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40464 req_id = b85a8889-3e78-4a3b-993c-1a931d35b64e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75172025-04-30T06:43:15.420ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1022 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40464 req_id = b85a8889-3e78-4a3b-993c-1a931d35b64e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75182025-04-30T06:43:15.426ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75192025-04-30T06:43:15.446ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75202025-04-30T06:43:15.446ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75212025-04-30T06:43:15.446ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
75222025-04-30T06:43:15.450ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75232025-04-30T06:43:15.450ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b85a8889-3e78-4a3b-993c-1a931d35b64e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
75242025-04-30T06:43:15.461ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75252025-04-30T06:43:15.461ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75262025-04-30T06:43:15.461ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
75272025-04-30T06:43:15.467ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75282025-04-30T06:43:15.467ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75292025-04-30T06:43:15.467ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75302025-04-30T06:43:15.468ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75312025-04-30T06:43:15.468ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75322025-04-30T06:43:15.471ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75332025-04-30T06:43:15.472ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
75342025-04-30T06:43:15.472ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75352025-04-30T06:43:15.475ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75362025-04-30T06:43:15.483ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75372025-04-30T06:43:15.485ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75382025-04-30T06:43:15.485ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75392025-04-30T06:43:15.486ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75402025-04-30T06:43:15.537ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
75412025-04-30T06:43:15.539ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
75422025-04-30T06:43:15.539ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 673.54811ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
75432025-04-30T06:43:15.540ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
75442025-04-30T06:43:15.540ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75452025-04-30T06:43:15.540ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75462025-04-30T06:43:15.540ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75472025-04-30T06:43:15.540ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75482025-04-30T06:43:15.543ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75492025-04-30T06:43:15.545ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
75502025-04-30T06:43:15.552ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75512025-04-30T06:43:15.566ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75522025-04-30T06:43:15.566ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75532025-04-30T06:43:15.566ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75542025-04-30T06:43:15.567ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75552025-04-30T06:43:15.572ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
75562025-04-30T06:43:15.572ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
75572025-04-30T06:43:15.573ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 89 5c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .\\..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61151 req_id = 80133c0a-999c-45f0-bbdf-2fcb60a72111
75582025-04-30T06:43:15.573ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35164,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61151 req_id = 80133c0a-999c-45f0-bbdf-2fcb60a72111
75592025-04-30T06:43:15.574ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
75602025-04-30T06:43:15.574ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75612025-04-30T06:43:15.574ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61151 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 80133c0a-999c-45f0-bbdf-2fcb60a72111
75622025-04-30T06:43:15.574ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 373661985 } })
75632025-04-30T06:43:15.574ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
75642025-04-30T06:43:15.575ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 45 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .E..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54267 req_id = ccbfcfeb-4bb0-4417-8269-97c68f689648
75652025-04-30T06:43:15.575ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33349,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54267 req_id = ccbfcfeb-4bb0-4417-8269-97c68f689648
75662025-04-30T06:43:15.575ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75672025-04-30T06:43:15.575ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54267 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ccbfcfeb-4bb0-4417-8269-97c68f689648
75682025-04-30T06:43:15.666ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
75692025-04-30T06:43:15.667ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75702025-04-30T06:43:15.667ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75712025-04-30T06:43:15.667ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75722025-04-30T06:43:15.667ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
75732025-04-30T06:43:15.669ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75742025-04-30T06:43:15.669ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
75752025-04-30T06:43:15.670ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75762025-04-30T06:43:15.672ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d61b910e-a5f1-4e43-88c5-0e1526ac18a3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75772025-04-30T06:43:15.672ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f6 87 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62484 req_id = be55b8d9-b20a-4d9a-9a28-0870de3eb990
75782025-04-30T06:43:15.672ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63111,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62484 req_id = be55b8d9-b20a-4d9a-9a28-0870de3eb990
75792025-04-30T06:43:15.673ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
75802025-04-30T06:43:15.673ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63111, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62484 req_id = be55b8d9-b20a-4d9a-9a28-0870de3eb990
75812025-04-30T06:43:15.674ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d61b910e-a5f1-4e43-88c5-0e1526ac18a3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75822025-04-30T06:43:15.674ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d61b910e-a5f1-4e43-88c5-0e1526ac18a3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75832025-04-30T06:43:15.676ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d61b910e-a5f1-4e43-88c5-0e1526ac18a3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75842025-04-30T06:43:15.677ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
75852025-04-30T06:43:15.677ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
75862025-04-30T06:43:15.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
75872025-04-30T06:43:15.678ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75882025-04-30T06:43:15.694ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75892025-04-30T06:43:15.694ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75902025-04-30T06:43:15.694ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
75912025-04-30T06:43:15.695ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75922025-04-30T06:43:15.696ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 23727 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d61b910e-a5f1-4e43-88c5-0e1526ac18a3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75932025-04-30T06:43:15.697ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "d61b910e-a5f1-4e43-88c5-0e1526ac18a3", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
75942025-04-30T06:43:15.697ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
75952025-04-30T06:43:15.698ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6d8c865c-61ea-4373-a781-35eda7e11896 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
75962025-04-30T06:43:15.700ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6d8c865c-61ea-4373-a781-35eda7e11896 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
75972025-04-30T06:43:15.700ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6d8c865c-61ea-4373-a781-35eda7e11896 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
75982025-04-30T06:43:15.701ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6d8c865c-61ea-4373-a781-35eda7e11896 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
75992025-04-30T06:43:15.701ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76002025-04-30T06:43:15.702ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
76012025-04-30T06:43:15.702ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76022025-04-30T06:43:15.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d8e1edd-baf5-4ca2-b5e2-59b63feff177", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76032025-04-30T06:43:15.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
76042025-04-30T06:43:15.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76052025-04-30T06:43:15.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76062025-04-30T06:43:15.706ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 7896 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6d8c865c-61ea-4373-a781-35eda7e11896 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
76072025-04-30T06:43:15.707ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d8c865c-61ea-4373-a781-35eda7e11896", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76082025-04-30T06:43:15.707ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76092025-04-30T06:43:15.707ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
76102025-04-30T06:43:15.707ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
76112025-04-30T06:43:15.709ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76122025-04-30T06:43:15.711ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76132025-04-30T06:43:15.711ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76142025-04-30T06:43:15.711ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76152025-04-30T06:43:15.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "434f0e79-5f50-4a25-86bc-01060994e8a5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76162025-04-30T06:43:15.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
76172025-04-30T06:43:15.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "791de63b-cf93-4003-aaaa-6d7bbbd02456", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76182025-04-30T06:43:15.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
76192025-04-30T06:43:15.719ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e98efbf5-9d8c-40c1-8991-0dc0b82e8eae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76202025-04-30T06:43:15.719ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
76212025-04-30T06:43:15.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "420690d5-21ef-41ac-a780-b2193af2bb16", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76222025-04-30T06:43:15.724ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
76232025-04-30T06:43:15.728ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d53e0ebf-9354-424a-aa6c-7bd6727716ff", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76242025-04-30T06:43:15.729ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
76252025-04-30T06:43:15.736ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd27868b-74d3-44a5-9fde-013904b9a7c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76262025-04-30T06:43:15.736ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
76272025-04-30T06:43:15.749ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
76282025-04-30T06:43:15.752ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d52efa55-8b6f-4a52-843d-cf0d28d4b265", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76292025-04-30T06:43:15.752ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
76302025-04-30T06:43:15.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5314b08-6847-4820-9f4e-6726f8af4584", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76312025-04-30T06:43:15.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
76322025-04-30T06:43:15.755ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c59e8297-68af-4683-935e-c18b4476eaa4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76332025-04-30T06:43:15.755ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
76342025-04-30T06:43:15.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78bfce42-efd4-45e0-9db7-47b98cb21feb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76352025-04-30T06:43:15.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
76362025-04-30T06:43:15.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e683e84d-79c0-4766-bd7b-fee8982d7cf4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76372025-04-30T06:43:15.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
76382025-04-30T06:43:15.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dbb14a5-9162-42e0-b96b-a21e2ccee54b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76392025-04-30T06:43:15.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
76402025-04-30T06:43:15.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0785511b-40f0-4029-8aab-2044070fce7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76412025-04-30T06:43:15.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
76422025-04-30T06:43:15.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0c7a397-ba92-4de3-8cc9-26ce653fbb94", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76432025-04-30T06:43:15.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
76442025-04-30T06:43:15.774ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84a1ccde-1c7a-4ad1-9139-34f3cb65ef3d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76452025-04-30T06:43:15.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
76462025-04-30T06:43:15.778ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02f2e59e-76cb-408b-9718-db3105cc0d07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76472025-04-30T06:43:15.778ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
76482025-04-30T06:43:15.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac5b44cf-d28c-4245-93ff-da46f039a300", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76492025-04-30T06:43:15.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
76502025-04-30T06:43:15.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "423d453e-6d83-4366-869e-7dddd48e12ee", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76512025-04-30T06:43:15.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
76522025-04-30T06:43:15.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e753285-695b-40c6-85d4-b1c767cd2632", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76532025-04-30T06:43:15.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
76542025-04-30T06:43:15.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7728ef91-d831-427c-ba22-de28c64bcf0c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76552025-04-30T06:43:15.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
76562025-04-30T06:43:15.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be0ad711-3e2b-4112-b561-ac7e708918d8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76572025-04-30T06:43:15.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
76582025-04-30T06:43:15.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb8ff10-97ff-4024-9922-0101cf81e371", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76592025-04-30T06:43:15.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
76602025-04-30T06:43:15.799ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76612025-04-30T06:43:15.799ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76622025-04-30T06:43:15.800ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76632025-04-30T06:43:15.800ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
76642025-04-30T06:43:15.801ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76652025-04-30T06:43:15.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cbfb5b7-67be-4b98-ae14-e6e33e001f55", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76662025-04-30T06:43:15.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
76672025-04-30T06:43:15.805ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76682025-04-30T06:43:15.813ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76692025-04-30T06:43:15.813ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76702025-04-30T06:43:15.814ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
76712025-04-30T06:43:15.814ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76722025-04-30T06:43:15.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "800db267-374a-4f4d-8f60-95849fea65f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76732025-04-30T06:43:15.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
76742025-04-30T06:43:15.817ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76752025-04-30T06:43:15.817ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
76762025-04-30T06:43:15.818ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76772025-04-30T06:43:15.821ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df2d02c-aa74-48d2-b16a-4c92e483c1ad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76782025-04-30T06:43:15.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
76792025-04-30T06:43:15.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22c0c613-8230-4d63-b148-ad011c7912d4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76802025-04-30T06:43:15.825ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
76812025-04-30T06:43:15.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac5fa67-b982-4227-a6a6-5930d4cd3813", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76822025-04-30T06:43:15.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
76832025-04-30T06:43:15.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d7a7408-2063-49a2-a58e-a747a55afafd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76842025-04-30T06:43:15.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
76852025-04-30T06:43:15.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75032e5d-5c68-444e-9a1f-fac348a82143", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76862025-04-30T06:43:15.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
76872025-04-30T06:43:15.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7fb0c02-44e6-4c99-a7c4-752162058d2f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76882025-04-30T06:43:15.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
76892025-04-30T06:43:15.844ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9c559f1-94a9-47e9-9a63-2e3bfa4836f0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76902025-04-30T06:43:15.845ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
76912025-04-30T06:43:15.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1132b99-760b-4db4-9afe-431817fd3cc2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76922025-04-30T06:43:15.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
76932025-04-30T06:43:15.847ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9633f29c-67e6-49d8-8017-553c58c1d936", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76942025-04-30T06:43:15.847ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
76952025-04-30T06:43:15.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "666fd82c-f9ba-4342-a773-534a19b7d88c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76962025-04-30T06:43:15.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
76972025-04-30T06:43:15.851ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6ab52ec-88f8-4619-97bc-946383d9fbba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
76982025-04-30T06:43:15.851ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
76992025-04-30T06:43:15.855ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
77002025-04-30T06:43:15.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df4be8a-507f-4531-b6e0-c10d6d98f5bc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77012025-04-30T06:43:15.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
77022025-04-30T06:43:15.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed189ee-397f-4ae5-bca1-c28d28784955", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77032025-04-30T06:43:15.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
77042025-04-30T06:43:15.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9850fa1-a1eb-49fd-a497-3733876c37a1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77052025-04-30T06:43:15.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
77062025-04-30T06:43:15.864ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ff61ea-94f4-4cf1-b065-92cdc4714a16", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77072025-04-30T06:43:15.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
77082025-04-30T06:43:15.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60c19288-2f91-4c9c-817d-b022bc4e99cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77092025-04-30T06:43:15.869ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
77102025-04-30T06:43:15.871ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f05ae72a-a54c-478b-bedb-e4695d8caf7f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77112025-04-30T06:43:15.871ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
77122025-04-30T06:43:15.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b7735d-075e-49cb-8627-f92798e1a4a2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77132025-04-30T06:43:15.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
77142025-04-30T06:43:15.873ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44cd535b-d29b-49de-8985-bfaf1a4bd6da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77152025-04-30T06:43:15.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
77162025-04-30T06:43:15.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a56511-5756-4b81-afc4-adf0b4e18f73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77172025-04-30T06:43:15.875ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77182025-04-30T06:43:15.875ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77192025-04-30T06:43:15.875ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77202025-04-30T06:43:15.875ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77212025-04-30T06:43:15.876ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77222025-04-30T06:43:15.880ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77232025-04-30T06:43:15.885ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77242025-04-30T06:43:15.885ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77252025-04-30T06:43:15.885ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77262025-04-30T06:43:15.886ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77272025-04-30T06:43:15.886ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77282025-04-30T06:43:15.886ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77292025-04-30T06:43:15.889ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77302025-04-30T06:43:15.893ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77312025-04-30T06:43:15.894ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77322025-04-30T06:43:15.894ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77332025-04-30T06:43:15.895ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77342025-04-30T06:43:15.961ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
77352025-04-30T06:43:15.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
77362025-04-30T06:43:15.961ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77372025-04-30T06:43:15.961ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77382025-04-30T06:43:15.961ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77392025-04-30T06:43:15.962ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
77402025-04-30T06:43:15.963ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77412025-04-30T06:43:15.963ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:64737 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
77422025-04-30T06:43:15.964ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:64737 req_id = 33bb36d8-ae62-4512-b451-f99b8a3815f9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77432025-04-30T06:43:15.964ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1057 local_addr = [::1]:45714 method = GET remote_addr = [::1]:64737 req_id = 33bb36d8-ae62-4512-b451-f99b8a3815f9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77442025-04-30T06:43:15.966ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77452025-04-30T06:43:15.975ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77462025-04-30T06:43:15.975ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77472025-04-30T06:43:15.975ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
77482025-04-30T06:43:15.975ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77492025-04-30T06:43:15.976ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "33bb36d8-ae62-4512-b451-f99b8a3815f9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:15 GMT"} })
77502025-04-30T06:43:15.979ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77512025-04-30T06:43:15.979ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77522025-04-30T06:43:15.979ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
77532025-04-30T06:43:15.983ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77542025-04-30T06:43:15.983ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77552025-04-30T06:43:15.983ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77562025-04-30T06:43:15.983ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77572025-04-30T06:43:15.984ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77582025-04-30T06:43:15.985ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77592025-04-30T06:43:15.985ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77602025-04-30T06:43:15.985ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
77612025-04-30T06:43:15.987ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
77622025-04-30T06:43:15.988ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77632025-04-30T06:43:15.988ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
77642025-04-30T06:43:15.990ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77652025-04-30T06:43:15.995ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
77662025-04-30T06:43:15.995ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
77672025-04-30T06:43:15.997ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77682025-04-30T06:43:16.005ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77692025-04-30T06:43:16.006ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77702025-04-30T06:43:16.006ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77712025-04-30T06:43:16.007ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77722025-04-30T06:43:16.029ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
77732025-04-30T06:43:16.029ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77742025-04-30T06:43:16.055ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77752025-04-30T06:43:16.055ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77762025-04-30T06:43:16.055ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 515.357034ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77772025-04-30T06:43:16.055ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77782025-04-30T06:43:16.055ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77792025-04-30T06:43:16.055ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77802025-04-30T06:43:16.055ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77812025-04-30T06:43:16.056ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77822025-04-30T06:43:16.057ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77832025-04-30T06:43:16.062ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77842025-04-30T06:43:16.068ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77852025-04-30T06:43:16.068ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77862025-04-30T06:43:16.068ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77872025-04-30T06:43:16.069ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77882025-04-30T06:43:16.071ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
77892025-04-30T06:43:16.075ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
77902025-04-30T06:43:16.075ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77912025-04-30T06:43:16.076ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a5 13 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55581 req_id = 02d7899f-053b-4e12-a7ae-26bb136f0024
77922025-04-30T06:43:16.076ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42259,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55581 req_id = 02d7899f-053b-4e12-a7ae-26bb136f0024
77932025-04-30T06:43:16.077ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
77942025-04-30T06:43:16.077ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77952025-04-30T06:43:16.077ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55581 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 02d7899f-053b-4e12-a7ae-26bb136f0024
77962025-04-30T06:43:16.078ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4260, tv_nsec: 877289208 } })
77972025-04-30T06:43:16.078ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
77982025-04-30T06:43:16.078ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 06 b9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51024 req_id = 80f3680a-e742-4524-a679-39b9b13921cc
77992025-04-30T06:43:16.079ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1721,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51024 req_id = 80f3680a-e742-4524-a679-39b9b13921cc
78002025-04-30T06:43:16.079ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78012025-04-30T06:43:16.079ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51024 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 80f3680a-e742-4524-a679-39b9b13921cc
78022025-04-30T06:43:16.175ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
78032025-04-30T06:43:16.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78042025-04-30T06:43:16.175ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78052025-04-30T06:43:16.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78062025-04-30T06:43:16.176ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78072025-04-30T06:43:16.176ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78082025-04-30T06:43:16.176ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78092025-04-30T06:43:16.177ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 66 85 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c f............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56405 req_id = 1e553a46-d1af-4e29-8377-59160566c976
78102025-04-30T06:43:16.177ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26245,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56405 req_id = 1e553a46-d1af-4e29-8377-59160566c976
78112025-04-30T06:43:16.177ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
78122025-04-30T06:43:16.177ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26245, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56405 req_id = 1e553a46-d1af-4e29-8377-59160566c976
78132025-04-30T06:43:16.178ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
78142025-04-30T06:43:16.178ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
78152025-04-30T06:43:16.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
78162025-04-30T06:43:16.182ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78172025-04-30T06:43:16.188ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78182025-04-30T06:43:16.188ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78192025-04-30T06:43:16.188ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78202025-04-30T06:43:16.189ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78212025-04-30T06:43:16.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2e96649-9cc8-4410-9b31-289c01abf5f8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78222025-04-30T06:43:16.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
78232025-04-30T06:43:16.194ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78242025-04-30T06:43:16.194ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78252025-04-30T06:43:16.194ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78262025-04-30T06:43:16.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a2574b0-e101-4886-9524-54cc0bc38ed5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78272025-04-30T06:43:16.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
78282025-04-30T06:43:16.198ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae6e1e5-7870-4142-b836-330a1198734f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78292025-04-30T06:43:16.199ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
78302025-04-30T06:43:16.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a93fca53-d216-4116-8c89-86a7ccede37e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78312025-04-30T06:43:16.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
78322025-04-30T06:43:16.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed144cb-11df-42e2-b62e-80bc21b6e4ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78332025-04-30T06:43:16.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
78342025-04-30T06:43:16.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d226545-c087-46b3-9e22-2731d00e599f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78352025-04-30T06:43:16.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
78362025-04-30T06:43:16.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5ee5a85-6278-486d-9906-e25fd0a76106", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78372025-04-30T06:43:16.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
78382025-04-30T06:43:16.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9765caab-7322-4891-9ec5-2e34c5f21a44", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78392025-04-30T06:43:16.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
78402025-04-30T06:43:16.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a869a41f-c741-4579-9268-c9a4794ce51c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78412025-04-30T06:43:16.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
78422025-04-30T06:43:16.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd088a9d-2e5c-467b-90e5-a50dbd24de2e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78432025-04-30T06:43:16.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
78442025-04-30T06:43:16.234ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
78452025-04-30T06:43:16.237ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aa7e6e0-d123-4edd-a69c-dc9b89d33620", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78462025-04-30T06:43:16.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
78472025-04-30T06:43:16.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7c746d-89be-4ddc-8692-5ee44982d138", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78482025-04-30T06:43:16.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
78492025-04-30T06:43:16.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4330994d-a4d6-4c89-8f56-8018547b77b7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78502025-04-30T06:43:16.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
78512025-04-30T06:43:16.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "274ad04d-8d4a-4b77-b707-5ced039f588a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78522025-04-30T06:43:16.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
78532025-04-30T06:43:16.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67dd0b7f-73cd-4949-b622-cb0a43c6ea3f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78542025-04-30T06:43:16.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
78552025-04-30T06:43:16.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c0db67-b9a5-4ffc-a920-e430de89b581", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78562025-04-30T06:43:16.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
78572025-04-30T06:43:16.275ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c53ded-98a8-4129-9409-d41d441ffe64", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78582025-04-30T06:43:16.275ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
78592025-04-30T06:43:16.284ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "793d73a8-7627-4127-a4f7-ef76eb2f3401", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78602025-04-30T06:43:16.286ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
78612025-04-30T06:43:16.286ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78622025-04-30T06:43:16.286ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78632025-04-30T06:43:16.286ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78642025-04-30T06:43:16.286ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78652025-04-30T06:43:16.287ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78662025-04-30T06:43:16.289ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df084ff1-1fa1-4339-a6da-88cde8ba4dab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78672025-04-30T06:43:16.290ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
78682025-04-30T06:43:16.291ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba8531c7-a964-447f-b3e8-a034d36765f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78692025-04-30T06:43:16.291ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
78702025-04-30T06:43:16.292ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78712025-04-30T06:43:16.300ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78722025-04-30T06:43:16.300ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78732025-04-30T06:43:16.300ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78742025-04-30T06:43:16.301ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78752025-04-30T06:43:16.303ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e664303-0a9e-498b-96c9-addc5c69c594", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78762025-04-30T06:43:16.303ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
78772025-04-30T06:43:16.304ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78782025-04-30T06:43:16.304ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
78792025-04-30T06:43:16.304ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78802025-04-30T06:43:16.304ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c5c5308-d14b-45ab-a998-dd90c146873d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78812025-04-30T06:43:16.304ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
78822025-04-30T06:43:16.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb928ff9-46c1-4baa-8a82-6cb57e1eb7ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78832025-04-30T06:43:16.309ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
78842025-04-30T06:43:16.311ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b67abb-602e-4a5e-8c45-1f6cef733883", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78852025-04-30T06:43:16.311ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
78862025-04-30T06:43:16.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02993606-1aa6-4dcc-92fe-3bfe0d0cfd17", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78872025-04-30T06:43:16.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
78882025-04-30T06:43:16.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59455efd-634e-48ea-a4d1-86a117367cfe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78892025-04-30T06:43:16.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
78902025-04-30T06:43:16.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf491283-d7e6-43a9-b7c8-284bd470401b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78912025-04-30T06:43:16.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
78922025-04-30T06:43:16.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "007f8842-0ef8-400c-b413-e42b07249870", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78932025-04-30T06:43:16.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
78942025-04-30T06:43:16.335ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
78952025-04-30T06:43:16.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "470ecaab-a580-4c65-83d8-2b28beac01e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78962025-04-30T06:43:16.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
78972025-04-30T06:43:16.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f805328b-9789-4f9e-891c-0d39491d6791", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
78982025-04-30T06:43:16.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
78992025-04-30T06:43:16.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c83ee8-2343-42a0-9908-3450bfcd7e1b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79002025-04-30T06:43:16.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
79012025-04-30T06:43:16.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaed1488-8ac1-4fcf-937c-115ef74b4e6b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79022025-04-30T06:43:16.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
79032025-04-30T06:43:16.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d383cc-4251-41e4-8a41-14c9443ba852", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79042025-04-30T06:43:16.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
79052025-04-30T06:43:16.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b03988b8-5c62-47d0-a018-20d2272b2b30", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79062025-04-30T06:43:16.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
79072025-04-30T06:43:16.354ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b00f53-1019-4bc4-90a3-bc6c4874d809", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79082025-04-30T06:43:16.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
79092025-04-30T06:43:16.357ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1db587b-8a2e-44e5-82de-887ddf95f534", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79102025-04-30T06:43:16.357ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
79112025-04-30T06:43:16.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f5f00c-486b-44fd-9682-3775b2ede5d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79122025-04-30T06:43:16.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
79132025-04-30T06:43:16.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab6e435a-28ae-4b80-8b28-540718f9d2d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79142025-04-30T06:43:16.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
79152025-04-30T06:43:16.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd6acbb8-6e23-4a24-8f76-0fff7553b41e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79162025-04-30T06:43:16.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
79172025-04-30T06:43:16.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a30fd8-131e-4a8f-8241-5ba87a857f6f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79182025-04-30T06:43:16.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
79192025-04-30T06:43:16.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f356b66-647f-4729-a106-84e5d03c471f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79202025-04-30T06:43:16.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
79212025-04-30T06:43:16.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e21fd91f-9d7d-4ac2-96c2-f7f71f2018d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79222025-04-30T06:43:16.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
79232025-04-30T06:43:16.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8671b3d6-6e9e-4689-a8f2-dac7be9386b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79242025-04-30T06:43:16.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
79252025-04-30T06:43:16.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd7b0a3-e56e-4122-83be-561416893d68", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79262025-04-30T06:43:16.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
79272025-04-30T06:43:16.374ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc469942-9164-40c7-aa51-04a7bdb4f9c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79282025-04-30T06:43:16.374ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79292025-04-30T06:43:16.375ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79302025-04-30T06:43:16.375ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79312025-04-30T06:43:16.375ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79322025-04-30T06:43:16.377ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79332025-04-30T06:43:16.393ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79342025-04-30T06:43:16.393ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79352025-04-30T06:43:16.393ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79362025-04-30T06:43:16.393ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
79372025-04-30T06:43:16.394ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79382025-04-30T06:43:16.397ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79392025-04-30T06:43:16.406ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79402025-04-30T06:43:16.406ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79412025-04-30T06:43:16.406ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
79422025-04-30T06:43:16.408ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79432025-04-30T06:43:16.408ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79442025-04-30T06:43:16.416ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79452025-04-30T06:43:16.416ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79462025-04-30T06:43:16.416ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79472025-04-30T06:43:16.416ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79482025-04-30T06:43:16.416ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79492025-04-30T06:43:16.417ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79502025-04-30T06:43:16.420ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79512025-04-30T06:43:16.420ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
79522025-04-30T06:43:16.420ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79532025-04-30T06:43:16.420ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79542025-04-30T06:43:16.428ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79552025-04-30T06:43:16.428ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79562025-04-30T06:43:16.428ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79572025-04-30T06:43:16.429ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79582025-04-30T06:43:16.509ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
79592025-04-30T06:43:16.509ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
79602025-04-30T06:43:16.510ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:43230 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79612025-04-30T06:43:16.510ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:43230 req_id = 1543ae35-636b-4713-aab4-3580da37d3ed sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79622025-04-30T06:43:16.512ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 887 local_addr = [::1]:45714 method = GET remote_addr = [::1]:43230 req_id = 1543ae35-636b-4713-aab4-3580da37d3ed response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79632025-04-30T06:43:16.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1543ae35-636b-4713-aab4-3580da37d3ed", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
79642025-04-30T06:43:16.516ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79652025-04-30T06:43:16.516ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79662025-04-30T06:43:16.516ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
79672025-04-30T06:43:16.519ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79682025-04-30T06:43:16.519ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79692025-04-30T06:43:16.519ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79702025-04-30T06:43:16.519ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79712025-04-30T06:43:16.520ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79722025-04-30T06:43:16.528ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79732025-04-30T06:43:16.533ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79742025-04-30T06:43:16.533ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79752025-04-30T06:43:16.533ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79762025-04-30T06:43:16.534ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79772025-04-30T06:43:16.555ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
79782025-04-30T06:43:16.561ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79792025-04-30T06:43:16.562ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79802025-04-30T06:43:16.562ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 506.436271ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79812025-04-30T06:43:16.562ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
79822025-04-30T06:43:16.562ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79832025-04-30T06:43:16.562ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79842025-04-30T06:43:16.562ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79852025-04-30T06:43:16.562ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79862025-04-30T06:43:16.562ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79872025-04-30T06:43:16.565ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79882025-04-30T06:43:16.571ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79892025-04-30T06:43:16.571ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79902025-04-30T06:43:16.571ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79912025-04-30T06:43:16.572ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79922025-04-30T06:43:16.575ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
79932025-04-30T06:43:16.576ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79942025-04-30T06:43:16.579ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 97 69 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .i..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59389 req_id = fdc4dd92-35c4-41ee-9aa6-135edc7a8c68
79952025-04-30T06:43:16.579ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38761,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59389 req_id = fdc4dd92-35c4-41ee-9aa6-135edc7a8c68
79962025-04-30T06:43:16.580ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
79972025-04-30T06:43:16.580ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79982025-04-30T06:43:16.580ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59389 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fdc4dd92-35c4-41ee-9aa6-135edc7a8c68
79992025-04-30T06:43:16.580ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4261, tv_nsec: 379684027 } })
80002025-04-30T06:43:16.580ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
80012025-04-30T06:43:16.581ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 51 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Q...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58284 req_id = dec7e2ea-017d-4c2c-8d2f-bfc12dfad187
80022025-04-30T06:43:16.581ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20760,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58284 req_id = dec7e2ea-017d-4c2c-8d2f-bfc12dfad187
80032025-04-30T06:43:16.581ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80042025-04-30T06:43:16.581ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58284 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dec7e2ea-017d-4c2c-8d2f-bfc12dfad187
80052025-04-30T06:43:16.650ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80062025-04-30T06:43:16.650ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80072025-04-30T06:43:16.650ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80082025-04-30T06:43:16.651ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80092025-04-30T06:43:16.651ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80102025-04-30T06:43:16.651ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80112025-04-30T06:43:16.652ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c0 38 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .8..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62378 req_id = 3ce6c049-616e-4ef9-9702-465df8740ae2
80122025-04-30T06:43:16.653ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49208,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62378 req_id = 3ce6c049-616e-4ef9-9702-465df8740ae2
80132025-04-30T06:43:16.653ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
80142025-04-30T06:43:16.653ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49208, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62378 req_id = 3ce6c049-616e-4ef9-9702-465df8740ae2
80152025-04-30T06:43:16.654ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
80162025-04-30T06:43:16.655ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
80172025-04-30T06:43:16.655ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
80182025-04-30T06:43:16.655ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80192025-04-30T06:43:16.660ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80202025-04-30T06:43:16.660ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80212025-04-30T06:43:16.660ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80222025-04-30T06:43:16.660ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80232025-04-30T06:43:16.662ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80242025-04-30T06:43:16.662ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80252025-04-30T06:43:16.663ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80262025-04-30T06:43:16.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dfae2b7-ee2a-4855-946d-21cc50c5da44", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80272025-04-30T06:43:16.665ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
80282025-04-30T06:43:16.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f8088da-dd3a-4a02-94dc-f18eef06ac18", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80292025-04-30T06:43:16.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
80302025-04-30T06:43:16.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55cfbd2a-e604-432d-bd7c-d062e7b11a50", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80312025-04-30T06:43:16.681ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
80322025-04-30T06:43:16.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "890bdf21-caba-4e6e-873e-748ee88dd4c7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80332025-04-30T06:43:16.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
80342025-04-30T06:43:16.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e2e8e98-febc-49fc-aaca-f97130d5e0d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80352025-04-30T06:43:16.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
80362025-04-30T06:43:16.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "470d0c98-5037-4901-a479-845b173e0535", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80372025-04-30T06:43:16.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
80382025-04-30T06:43:16.699ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
80392025-04-30T06:43:16.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80402025-04-30T06:43:16.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80412025-04-30T06:43:16.706ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80422025-04-30T06:43:16.707ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80432025-04-30T06:43:16.707ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80442025-04-30T06:43:16.707ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80452025-04-30T06:43:16.708ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e805f6c2-791b-464b-a552-5e93d0aafbdd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80462025-04-30T06:43:16.708ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
80472025-04-30T06:43:16.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f1394b9-8ead-4316-9db0-81d189b04235", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80482025-04-30T06:43:16.712ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
80492025-04-30T06:43:16.714ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64b2ab98-eb03-46c9-a248-beeb8ff0cda3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80502025-04-30T06:43:16.714ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
80512025-04-30T06:43:16.715ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26a8b6f-3b3b-4dc5-8ace-b84a58df4569", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80522025-04-30T06:43:16.715ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
80532025-04-30T06:43:16.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64b25d1-8756-45a4-9135-bfc9c346bc6d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80542025-04-30T06:43:16.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
80552025-04-30T06:43:16.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5a8e7cc-c081-4a25-a536-2915f6f17ca2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80562025-04-30T06:43:16.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
80572025-04-30T06:43:16.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0719489e-7e5a-4eeb-99f1-a2aa0eb24b4c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80582025-04-30T06:43:16.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
80592025-04-30T06:43:16.724ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eeb0ec6-87c4-4005-afdc-ae396ac8a5e0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80602025-04-30T06:43:16.724ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
80612025-04-30T06:43:16.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9df0f9c-5d41-4796-8c73-e3e2b4ec4b6d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80622025-04-30T06:43:16.727ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
80632025-04-30T06:43:16.729ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7262221-9cb9-493f-b65b-4ab04b0c7db9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80642025-04-30T06:43:16.729ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
80652025-04-30T06:43:16.732ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec34beda-5175-452d-9ae9-8c06a74407f6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80662025-04-30T06:43:16.732ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
80672025-04-30T06:43:16.741ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bef3321-7a5c-4b2c-ab1b-f93b95095c55", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80682025-04-30T06:43:16.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
80692025-04-30T06:43:16.750ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80702025-04-30T06:43:16.750ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80712025-04-30T06:43:16.751ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80722025-04-30T06:43:16.751ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80732025-04-30T06:43:16.752ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80742025-04-30T06:43:16.755ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b40b5e1-3a86-497e-bfea-99f2d8ceb793", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80752025-04-30T06:43:16.755ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
80762025-04-30T06:43:16.755ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80772025-04-30T06:43:16.761ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80782025-04-30T06:43:16.761ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80792025-04-30T06:43:16.761ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80802025-04-30T06:43:16.762ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80812025-04-30T06:43:16.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff8ab73b-0a77-4a4b-abef-87c3fb4324b8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80822025-04-30T06:43:16.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
80832025-04-30T06:43:16.766ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80842025-04-30T06:43:16.766ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
80852025-04-30T06:43:16.766ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80862025-04-30T06:43:16.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c987ef-ec84-44ae-9cc3-ad82956b4428", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80872025-04-30T06:43:16.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
80882025-04-30T06:43:16.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f52a1097-fe32-4174-81f9-293e8d5c5ec8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80892025-04-30T06:43:16.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
80902025-04-30T06:43:16.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d03f05c-0bd8-4cdc-9cac-443233ec0a10", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80912025-04-30T06:43:16.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
80922025-04-30T06:43:16.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ba99e2-bd30-4895-aaf5-ad71967d9324", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80932025-04-30T06:43:16.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
80942025-04-30T06:43:16.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56805cfd-9347-43ef-83dc-55c9fde64ee1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80952025-04-30T06:43:16.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
80962025-04-30T06:43:16.797ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aca52e70-e011-4b61-8834-4c464ac2042a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
80972025-04-30T06:43:16.797ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
80982025-04-30T06:43:16.802ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
80992025-04-30T06:43:16.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6fe68c9-3036-4a8e-b476-05c35083a934", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81002025-04-30T06:43:16.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
81012025-04-30T06:43:16.805ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35592891-e6a0-4998-9239-fe25ca8e3c1e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81022025-04-30T06:43:16.805ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
81032025-04-30T06:43:16.808ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc33243-b872-43e1-b54e-12d957b6c33d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81042025-04-30T06:43:16.808ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
81052025-04-30T06:43:16.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc6716a-ad3e-4dbd-b082-0c0ee5eed9d2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81062025-04-30T06:43:16.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
81072025-04-30T06:43:16.811ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cee4d363-03bd-44b5-9efb-32798535795f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81082025-04-30T06:43:16.811ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
81092025-04-30T06:43:16.812ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "204de025-d36d-4a41-bcae-6e5e3ecc12e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81102025-04-30T06:43:16.812ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
81112025-04-30T06:43:16.814ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2143eb5-3d40-42b5-aa62-932fa99d78bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81122025-04-30T06:43:16.814ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
81132025-04-30T06:43:16.815ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c413756b-aaab-4554-bbc9-7adbbf483ed5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81142025-04-30T06:43:16.815ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
81152025-04-30T06:43:16.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c81af70-48bc-4bb1-971a-893362a8acd7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81162025-04-30T06:43:16.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
81172025-04-30T06:43:16.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "839621c5-37f5-43a6-899d-2b058ab8e55b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81182025-04-30T06:43:16.820ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
81192025-04-30T06:43:16.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad9e726-201f-4b73-a351-22d8c2372473", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81202025-04-30T06:43:16.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
81212025-04-30T06:43:16.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee97fb7-dd45-461b-99d0-5d6c90b0c389", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81222025-04-30T06:43:16.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
81232025-04-30T06:43:16.826ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "994e929c-978b-43d0-88f7-66b51ea20cfa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81242025-04-30T06:43:16.826ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
81252025-04-30T06:43:16.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbee8c6e-92e9-4411-88b8-7330654f3eff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81262025-04-30T06:43:16.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
81272025-04-30T06:43:16.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f29443-187a-4956-b335-2f9fb4716fc2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81282025-04-30T06:43:16.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
81292025-04-30T06:43:16.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40d4e031-d085-483d-a339-655eab5ff80b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81302025-04-30T06:43:16.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
81312025-04-30T06:43:16.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3822019a-06f5-4bc2-8077-9e74a5547d13", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81322025-04-30T06:43:16.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
81332025-04-30T06:43:16.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "886081f7-d438-4cfe-bede-e0b7fb766c17", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81342025-04-30T06:43:16.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
81352025-04-30T06:43:16.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17942a4-75b9-40dc-ac2d-b074b992acb2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81362025-04-30T06:43:16.838ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81372025-04-30T06:43:16.838ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81382025-04-30T06:43:16.838ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81392025-04-30T06:43:16.838ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81402025-04-30T06:43:16.838ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81412025-04-30T06:43:16.843ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81422025-04-30T06:43:16.852ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81432025-04-30T06:43:16.852ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81442025-04-30T06:43:16.852ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81452025-04-30T06:43:16.852ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81462025-04-30T06:43:16.852ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81472025-04-30T06:43:16.853ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81482025-04-30T06:43:16.857ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81492025-04-30T06:43:16.857ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81502025-04-30T06:43:16.857ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81512025-04-30T06:43:16.857ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
81522025-04-30T06:43:16.858ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81532025-04-30T06:43:16.859ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81542025-04-30T06:43:16.864ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81552025-04-30T06:43:16.864ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81562025-04-30T06:43:16.864ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81572025-04-30T06:43:16.866ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81582025-04-30T06:43:16.869ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81592025-04-30T06:43:16.882ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81602025-04-30T06:43:16.882ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81612025-04-30T06:43:16.882ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
81622025-04-30T06:43:16.883ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81632025-04-30T06:43:16.960ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
81642025-04-30T06:43:16.960ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
81652025-04-30T06:43:16.961ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40676 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81662025-04-30T06:43:16.962ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81672025-04-30T06:43:16.962ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
81682025-04-30T06:43:16.964ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81692025-04-30T06:43:16.964ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40676 req_id = 0d646bff-446a-4c2c-a0d7-03e01156c48d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81702025-04-30T06:43:16.964ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 555 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40676 req_id = 0d646bff-446a-4c2c-a0d7-03e01156c48d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81712025-04-30T06:43:16.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d646bff-446a-4c2c-a0d7-03e01156c48d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:16 GMT"} })
81722025-04-30T06:43:16.973ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81732025-04-30T06:43:16.973ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81742025-04-30T06:43:16.973ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
81752025-04-30T06:43:16.975ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81762025-04-30T06:43:16.975ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81772025-04-30T06:43:16.975ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81782025-04-30T06:43:16.975ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81792025-04-30T06:43:16.976ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81802025-04-30T06:43:16.978ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81812025-04-30T06:43:16.980ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81822025-04-30T06:43:16.980ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
81832025-04-30T06:43:16.981ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
81842025-04-30T06:43:16.984ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81852025-04-30T06:43:16.990ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81862025-04-30T06:43:16.990ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81872025-04-30T06:43:16.990ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81882025-04-30T06:43:16.993ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81892025-04-30T06:43:16.994ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
81902025-04-30T06:43:16.994ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
81912025-04-30T06:43:17.005ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
81922025-04-30T06:43:17.005ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81932025-04-30T06:43:17.047ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81942025-04-30T06:43:17.047ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81952025-04-30T06:43:17.047ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 485.292232ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81962025-04-30T06:43:17.047ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
81972025-04-30T06:43:17.047ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81982025-04-30T06:43:17.047ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81992025-04-30T06:43:17.047ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82002025-04-30T06:43:17.047ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82012025-04-30T06:43:17.048ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82022025-04-30T06:43:17.062ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
82032025-04-30T06:43:17.062ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82042025-04-30T06:43:17.066ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82052025-04-30T06:43:17.066ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82062025-04-30T06:43:17.066ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82072025-04-30T06:43:17.067ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82082025-04-30T06:43:17.085ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82092025-04-30T06:43:17.085ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
82102025-04-30T06:43:17.085ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82112025-04-30T06:43:17.086ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8b 00 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64822 req_id = 7d242eee-551f-4a23-9625-a56e3304e4c7
82122025-04-30T06:43:17.088ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35584,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64822 req_id = 7d242eee-551f-4a23-9625-a56e3304e4c7
82132025-04-30T06:43:17.089ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
82142025-04-30T06:43:17.089ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82152025-04-30T06:43:17.089ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64822 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7d242eee-551f-4a23-9625-a56e3304e4c7
82162025-04-30T06:43:17.089ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4261, tv_nsec: 888612934 } })
82172025-04-30T06:43:17.089ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
82182025-04-30T06:43:17.090ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ba a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60626 req_id = 35f07aba-ccde-49c7-a0bb-d122cc82fb80
82192025-04-30T06:43:17.090ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47785,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60626 req_id = 35f07aba-ccde-49c7-a0bb-d122cc82fb80
82202025-04-30T06:43:17.090ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82212025-04-30T06:43:17.090ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60626 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 35f07aba-ccde-49c7-a0bb-d122cc82fb80
82222025-04-30T06:43:17.173ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82232025-04-30T06:43:17.173ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82242025-04-30T06:43:17.173ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82252025-04-30T06:43:17.173ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82262025-04-30T06:43:17.173ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
82272025-04-30T06:43:17.176ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82282025-04-30T06:43:17.177ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a8 2c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .,..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65517 req_id = 54087309-35b5-47eb-ba15-5d07648baeca
82292025-04-30T06:43:17.177ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43052,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65517 req_id = 54087309-35b5-47eb-ba15-5d07648baeca
82302025-04-30T06:43:17.177ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
82312025-04-30T06:43:17.177ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65517 req_id = 54087309-35b5-47eb-ba15-5d07648baeca
82322025-04-30T06:43:17.177ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
82332025-04-30T06:43:17.177ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
82342025-04-30T06:43:17.177ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
82352025-04-30T06:43:17.179ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82362025-04-30T06:43:17.185ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82372025-04-30T06:43:17.185ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82382025-04-30T06:43:17.185ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
82392025-04-30T06:43:17.186ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82402025-04-30T06:43:17.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "54b6b6b1-b823-4a6d-a764-aed4020eb970", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82412025-04-30T06:43:17.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
82422025-04-30T06:43:17.192ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82432025-04-30T06:43:17.193ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
82442025-04-30T06:43:17.194ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82452025-04-30T06:43:17.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec61611d-cf4e-4d08-92bd-043a624599a3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82462025-04-30T06:43:17.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
82472025-04-30T06:43:17.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a695bc-326d-45c9-88da-b7850ed59053", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82482025-04-30T06:43:17.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
82492025-04-30T06:43:17.203ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "375df069-f844-456c-82f7-58d160e31461", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82502025-04-30T06:43:17.203ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
82512025-04-30T06:43:17.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44159ad1-f0c5-40b0-be89-32e06224a0a0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82522025-04-30T06:43:17.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
82532025-04-30T06:43:17.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb085779-ba21-4f50-8aea-b62eca2bb621", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82542025-04-30T06:43:17.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
82552025-04-30T06:43:17.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42e49cd3-0071-43fa-a008-fba2b66d107a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82562025-04-30T06:43:17.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
82572025-04-30T06:43:17.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35bbfbf6-694c-41d2-a4fc-dd5a8f3138b4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82582025-04-30T06:43:17.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
82592025-04-30T06:43:17.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "254a9de8-1c05-42e6-92b6-02e7b6c60172", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82602025-04-30T06:43:17.224ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
82612025-04-30T06:43:17.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be32ade9-494e-4af7-971d-a31471d987a1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82622025-04-30T06:43:17.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
82632025-04-30T06:43:17.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a9dd80c-f8d0-42b6-b1ab-79814b733086", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82642025-04-30T06:43:17.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
82652025-04-30T06:43:17.236ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
82662025-04-30T06:43:17.238ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2d509fa-728d-41fd-a322-6d1d9bbf6508", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82672025-04-30T06:43:17.238ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
82682025-04-30T06:43:17.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "970ac672-f61a-44bf-a130-391f9fa3fe57", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82692025-04-30T06:43:17.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
82702025-04-30T06:43:17.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de3b9950-b268-43fb-9f92-a039daa0c782", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82712025-04-30T06:43:17.242ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
82722025-04-30T06:43:17.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afab17dc-8f96-41ea-86da-8d43ec44605c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82732025-04-30T06:43:17.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
82742025-04-30T06:43:17.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f59ebd0c-4c7e-4ee3-9feb-3dcfc97c58da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82752025-04-30T06:43:17.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
82762025-04-30T06:43:17.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39e564d1-007c-47f8-a621-1a70cc2ddca7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82772025-04-30T06:43:17.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
82782025-04-30T06:43:17.274ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f05810c-78e0-4b0e-a50c-ec3b0695dc08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82792025-04-30T06:43:17.274ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
82802025-04-30T06:43:17.285ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12744391-0f34-4f69-bf7b-60ea4b841fdc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82812025-04-30T06:43:17.285ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
82822025-04-30T06:43:17.286ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11a6bb3-7ec0-46f2-bebd-3779d6d810c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82832025-04-30T06:43:17.287ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
82842025-04-30T06:43:17.287ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82852025-04-30T06:43:17.287ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82862025-04-30T06:43:17.287ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82872025-04-30T06:43:17.287ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
82882025-04-30T06:43:17.288ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82892025-04-30T06:43:17.289ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bf1f07-efab-4585-b951-8bed3185e86f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82902025-04-30T06:43:17.289ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
82912025-04-30T06:43:17.290ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd18881c-a0ec-408d-83ea-55490a534b02", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82922025-04-30T06:43:17.291ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
82932025-04-30T06:43:17.291ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82942025-04-30T06:43:17.298ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82952025-04-30T06:43:17.298ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82962025-04-30T06:43:17.298ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
82972025-04-30T06:43:17.299ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82982025-04-30T06:43:17.300ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98215278-8e18-4c02-84ed-0c090a56faf6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
82992025-04-30T06:43:17.301ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
83002025-04-30T06:43:17.302ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f9977e-07e9-49b0-90bf-1892882af132", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83012025-04-30T06:43:17.302ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
83022025-04-30T06:43:17.303ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83032025-04-30T06:43:17.303ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
83042025-04-30T06:43:17.306ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83052025-04-30T06:43:17.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f6080f-17e2-487c-8038-dec49d321f83", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83062025-04-30T06:43:17.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
83072025-04-30T06:43:17.317ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c17be0e7-aefd-48b1-94fd-3149f63464f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83082025-04-30T06:43:17.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
83092025-04-30T06:43:17.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d9ee65-8353-4454-adfb-ae6e01f5d318", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83102025-04-30T06:43:17.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
83112025-04-30T06:43:17.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "376c1a62-261f-425e-b25b-091eb3570ae7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83122025-04-30T06:43:17.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
83132025-04-30T06:43:17.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d30f67e-41b5-42f5-ac32-9d46a7687488", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83142025-04-30T06:43:17.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
83152025-04-30T06:43:17.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03ce9a18-0dfc-443d-bea2-8af2620717a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83162025-04-30T06:43:17.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
83172025-04-30T06:43:17.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "612536b8-f1e3-4112-be61-fb71bd836fce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83182025-04-30T06:43:17.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
83192025-04-30T06:43:17.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0acf80ba-5a8e-447d-b617-dbee5bed5a67", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83202025-04-30T06:43:17.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
83212025-04-30T06:43:17.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a21648-ae45-441f-9062-58020109777e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83222025-04-30T06:43:17.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
83232025-04-30T06:43:17.334ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda71a1a-ad35-4404-884d-eb0bb9e66fc8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83242025-04-30T06:43:17.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
83252025-04-30T06:43:17.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "209fa209-6b15-4d49-b2c8-60b71b5cbad6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83262025-04-30T06:43:17.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
83272025-04-30T06:43:17.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf8fb58-d0e2-4320-9847-3ef30f7b98f5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83282025-04-30T06:43:17.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
83292025-04-30T06:43:17.342ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
83302025-04-30T06:43:17.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfa05859-3b8c-44bf-8956-39919fe201e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83312025-04-30T06:43:17.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
83322025-04-30T06:43:17.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f0c10b0-b18c-4641-9d29-58f54f73491a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83332025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
83342025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e4489e-1f0f-489a-9b9d-29a698d39eb5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83352025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
83362025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63bd69a6-36db-4fb9-9a35-e79034887519", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83372025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
83382025-04-30T06:43:17.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19be6af9-d546-48cd-9bf8-dcbaa9f24d8d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83392025-04-30T06:43:17.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
83402025-04-30T06:43:17.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "075785f1-7693-4442-ac57-4423fced616a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83412025-04-30T06:43:17.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
83422025-04-30T06:43:17.352ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3d6e36-56f9-4010-8c2d-26e62d0ed6b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83432025-04-30T06:43:17.352ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
83442025-04-30T06:43:17.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ddfd91-2e0f-4740-81df-53daaf1f93ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83452025-04-30T06:43:17.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
83462025-04-30T06:43:17.354ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c65899-990b-40d5-ac27-3a00eb04cbe5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83472025-04-30T06:43:17.354ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83482025-04-30T06:43:17.354ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83492025-04-30T06:43:17.354ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83502025-04-30T06:43:17.354ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83512025-04-30T06:43:17.355ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83522025-04-30T06:43:17.357ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83532025-04-30T06:43:17.364ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83542025-04-30T06:43:17.364ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83552025-04-30T06:43:17.364ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83562025-04-30T06:43:17.364ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83572025-04-30T06:43:17.364ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83582025-04-30T06:43:17.365ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83592025-04-30T06:43:17.381ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83602025-04-30T06:43:17.389ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83612025-04-30T06:43:17.390ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83622025-04-30T06:43:17.390ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83632025-04-30T06:43:17.390ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83642025-04-30T06:43:17.395ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83652025-04-30T06:43:17.396ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83662025-04-30T06:43:17.397ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83672025-04-30T06:43:17.397ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
83682025-04-30T06:43:17.398ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83692025-04-30T06:43:17.512ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
83702025-04-30T06:43:17.512ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
83712025-04-30T06:43:17.513ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:42345 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83722025-04-30T06:43:17.514ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83732025-04-30T06:43:17.528ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83742025-04-30T06:43:17.529ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83752025-04-30T06:43:17.529ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
83762025-04-30T06:43:17.530ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83772025-04-30T06:43:17.531ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:42345 req_id = 0eadb5f1-bcb3-4b51-8565-cf018044d69a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83782025-04-30T06:43:17.532ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1745 local_addr = [::1]:45714 method = GET remote_addr = [::1]:42345 req_id = 0eadb5f1-bcb3-4b51-8565-cf018044d69a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83792025-04-30T06:43:17.534ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eadb5f1-bcb3-4b51-8565-cf018044d69a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
83802025-04-30T06:43:17.539ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83812025-04-30T06:43:17.540ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83822025-04-30T06:43:17.540ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
83832025-04-30T06:43:17.544ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83842025-04-30T06:43:17.544ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83852025-04-30T06:43:17.544ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83862025-04-30T06:43:17.544ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83872025-04-30T06:43:17.545ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83882025-04-30T06:43:17.546ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83892025-04-30T06:43:17.547ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
83902025-04-30T06:43:17.547ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83912025-04-30T06:43:17.549ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83922025-04-30T06:43:17.556ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83932025-04-30T06:43:17.556ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83942025-04-30T06:43:17.556ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83952025-04-30T06:43:17.557ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83962025-04-30T06:43:17.594ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
83972025-04-30T06:43:17.595ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
83982025-04-30T06:43:17.596ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
83992025-04-30T06:43:17.597ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 549.489672ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84002025-04-30T06:43:17.597ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84012025-04-30T06:43:17.597ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84022025-04-30T06:43:17.597ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84032025-04-30T06:43:17.597ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84042025-04-30T06:43:17.597ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84052025-04-30T06:43:17.598ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84062025-04-30T06:43:17.601ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84072025-04-30T06:43:17.610ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84082025-04-30T06:43:17.610ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84092025-04-30T06:43:17.610ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84102025-04-30T06:43:17.613ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84112025-04-30T06:43:17.620ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
84122025-04-30T06:43:17.621ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84132025-04-30T06:43:17.623ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e5 5a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .Z..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61006 req_id = 4a91e19f-0a03-4b81-a798-c8fba940af48
84142025-04-30T06:43:17.623ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58714,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61006 req_id = 4a91e19f-0a03-4b81-a798-c8fba940af48
84152025-04-30T06:43:17.623ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
84162025-04-30T06:43:17.625ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84172025-04-30T06:43:17.625ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61006 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4a91e19f-0a03-4b81-a798-c8fba940af48
84182025-04-30T06:43:17.625ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 424651406 } })
84192025-04-30T06:43:17.625ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
84202025-04-30T06:43:17.626ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2d dc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e -...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56457 req_id = 3b083975-4f49-43a5-bbf3-65d468149d98
84212025-04-30T06:43:17.626ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11740,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56457 req_id = 3b083975-4f49-43a5-bbf3-65d468149d98
84222025-04-30T06:43:17.627ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84232025-04-30T06:43:17.627ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56457 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3b083975-4f49-43a5-bbf3-65d468149d98
84242025-04-30T06:43:17.698ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84252025-04-30T06:43:17.699ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84262025-04-30T06:43:17.699ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84272025-04-30T06:43:17.699ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84282025-04-30T06:43:17.699ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
84292025-04-30T06:43:17.700ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84302025-04-30T06:43:17.700ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
84312025-04-30T06:43:17.701ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84322025-04-30T06:43:17.702ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5ad11edc-6486-413f-bd5e-827a091d4241 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84332025-04-30T06:43:17.702ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 11 38 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .8..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54958 req_id = b1ab66d7-c52c-488e-9eca-f321664088ba
84342025-04-30T06:43:17.702ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4408,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54958 req_id = b1ab66d7-c52c-488e-9eca-f321664088ba
84352025-04-30T06:43:17.703ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
84362025-04-30T06:43:17.703ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4408, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54958 req_id = b1ab66d7-c52c-488e-9eca-f321664088ba
84372025-04-30T06:43:17.704ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5ad11edc-6486-413f-bd5e-827a091d4241 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84382025-04-30T06:43:17.704ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5ad11edc-6486-413f-bd5e-827a091d4241 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84392025-04-30T06:43:17.705ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5ad11edc-6486-413f-bd5e-827a091d4241 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84402025-04-30T06:43:17.705ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
84412025-04-30T06:43:17.706ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
84422025-04-30T06:43:17.706ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
84432025-04-30T06:43:17.706ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84442025-04-30T06:43:17.706ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84452025-04-30T06:43:17.708ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84462025-04-30T06:43:17.715ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84472025-04-30T06:43:17.715ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84482025-04-30T06:43:17.715ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
84492025-04-30T06:43:17.716ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84502025-04-30T06:43:17.717ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84512025-04-30T06:43:17.718ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84522025-04-30T06:43:17.718ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84532025-04-30T06:43:17.718ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84542025-04-30T06:43:17.719ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 16940 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 5ad11edc-6486-413f-bd5e-827a091d4241 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
84552025-04-30T06:43:17.720ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad11edc-6486-413f-bd5e-827a091d4241", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84562025-04-30T06:43:17.729ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84572025-04-30T06:43:17.730ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84582025-04-30T06:43:17.730ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
84592025-04-30T06:43:17.731ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84602025-04-30T06:43:17.731ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2506306e-7a33-4098-be98-e019ca2c17e7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84612025-04-30T06:43:17.732ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2506306e-7a33-4098-be98-e019ca2c17e7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84622025-04-30T06:43:17.732ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2506306e-7a33-4098-be98-e019ca2c17e7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84632025-04-30T06:43:17.733ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2506306e-7a33-4098-be98-e019ca2c17e7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84642025-04-30T06:43:17.735ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b94b201-dd89-4525-9fd0-b7817d878a2e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84652025-04-30T06:43:17.736ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
84662025-04-30T06:43:17.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b60022-d9a2-4ec5-93b6-4d27c7e67f35", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84672025-04-30T06:43:17.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
84682025-04-30T06:43:17.739ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 8226 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2506306e-7a33-4098-be98-e019ca2c17e7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
84692025-04-30T06:43:17.740ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2506306e-7a33-4098-be98-e019ca2c17e7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84702025-04-30T06:43:17.741ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84712025-04-30T06:43:17.741ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
84722025-04-30T06:43:17.741ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
84732025-04-30T06:43:17.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a229c0-444f-46c4-a887-d297464ac82a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84742025-04-30T06:43:17.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
84752025-04-30T06:43:17.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f39839-1131-4414-a2b3-f11014f506cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84762025-04-30T06:43:17.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
84772025-04-30T06:43:17.751ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d059063d-a7fc-428c-a864-f3d507027f3c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84782025-04-30T06:43:17.751ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
84792025-04-30T06:43:17.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c4cf9a0-adee-4df8-99ab-006984ff5bab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84802025-04-30T06:43:17.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
84812025-04-30T06:43:17.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6f94362-3ffa-4159-9f33-b07dd15f0f9c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84822025-04-30T06:43:17.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
84832025-04-30T06:43:17.762ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21dbc883-6cab-45cc-b490-01ba2e26fc83", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84842025-04-30T06:43:17.762ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
84852025-04-30T06:43:17.763ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
84862025-04-30T06:43:17.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef9b12bc-da1a-4325-aece-e8f0858191e0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84872025-04-30T06:43:17.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
84882025-04-30T06:43:17.765ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "747e703a-6078-4b20-b7af-0c21ac0ec679", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84892025-04-30T06:43:17.765ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
84902025-04-30T06:43:17.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb08891-c52a-4927-a06b-5d63d0d82944", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84912025-04-30T06:43:17.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
84922025-04-30T06:43:17.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a063a66-903b-4795-a0c1-747f8ec96ad0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84932025-04-30T06:43:17.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
84942025-04-30T06:43:17.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0458ed9-11eb-4a7d-b4d1-b43721fb70f2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84952025-04-30T06:43:17.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
84962025-04-30T06:43:17.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4313d4d-2369-464a-bc94-0378b430a137", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84972025-04-30T06:43:17.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
84982025-04-30T06:43:17.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "144eee6a-e378-4837-9958-c61c1573ce23", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
84992025-04-30T06:43:17.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
85002025-04-30T06:43:17.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32afdc3e-4d8f-466b-9dbb-b82271dfcd75", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85012025-04-30T06:43:17.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
85022025-04-30T06:43:17.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22242cb6-7f1c-4c63-b4b5-80c529f1f21e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85032025-04-30T06:43:17.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
85042025-04-30T06:43:17.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c113e887-d4d1-4941-9b82-6f09210a3b6c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85052025-04-30T06:43:17.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
85062025-04-30T06:43:17.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10a0119-f380-4fcd-9793-af54fb865e88", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85072025-04-30T06:43:17.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
85082025-04-30T06:43:17.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0119d3a6-1169-4c04-9cd6-94fd93451655", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85092025-04-30T06:43:17.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
85102025-04-30T06:43:17.785ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8cf1075-67a9-444f-a7b2-411644e60b59", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85112025-04-30T06:43:17.785ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
85122025-04-30T06:43:17.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3d75542-dd59-4ada-bf17-d47f21323912", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85132025-04-30T06:43:17.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
85142025-04-30T06:43:17.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b168c1f0-be4d-4317-b7f6-b21f812ca7b3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85152025-04-30T06:43:17.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
85162025-04-30T06:43:17.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c5d5ecc-39e0-4335-8673-447e49c9d8b5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85172025-04-30T06:43:17.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
85182025-04-30T06:43:17.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f946745d-2bd2-462b-8a5b-68353a65ac5a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85192025-04-30T06:43:17.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
85202025-04-30T06:43:17.797ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8bed1ec-b1b5-4a60-b9db-ba98ec45a605", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85212025-04-30T06:43:17.805ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
85222025-04-30T06:43:17.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea6a514-2eba-4ccf-9ebf-7527d25fd24b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85232025-04-30T06:43:17.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
85242025-04-30T06:43:17.813ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34979f54-2724-48a9-a933-c2674da0151a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85252025-04-30T06:43:17.813ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
85262025-04-30T06:43:17.814ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85272025-04-30T06:43:17.814ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85282025-04-30T06:43:17.814ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85292025-04-30T06:43:17.814ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
85302025-04-30T06:43:17.815ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85312025-04-30T06:43:17.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "263c013f-e933-4a40-8a2f-ca9923e1434e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85322025-04-30T06:43:17.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
85332025-04-30T06:43:17.817ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85342025-04-30T06:43:17.823ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85352025-04-30T06:43:17.823ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85362025-04-30T06:43:17.823ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
85372025-04-30T06:43:17.824ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85382025-04-30T06:43:17.827ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85392025-04-30T06:43:17.827ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
85402025-04-30T06:43:17.827ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85412025-04-30T06:43:17.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98adf284-9bce-4fc5-b099-65628963f5c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85422025-04-30T06:43:17.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
85432025-04-30T06:43:17.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c17fcf0d-50d7-47e1-b53d-b15edeeac5fa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85442025-04-30T06:43:17.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
85452025-04-30T06:43:17.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b17ed89-2e56-45db-a0b1-ce320aa44a09", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85462025-04-30T06:43:17.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
85472025-04-30T06:43:17.833ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27adb043-5021-4798-9d1c-63f188111b62", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85482025-04-30T06:43:17.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
85492025-04-30T06:43:17.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9c36621-415b-4664-9b5d-1d56145e1784", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85502025-04-30T06:43:17.836ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
85512025-04-30T06:43:17.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dccb0a5f-0e38-411e-965a-6798ffc799c7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85522025-04-30T06:43:17.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
85532025-04-30T06:43:17.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42850b93-0b1d-4737-8bf7-4181807ad143", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85542025-04-30T06:43:17.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
85552025-04-30T06:43:17.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c6bf3a-e912-4356-8f4f-16aa6674de96", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85562025-04-30T06:43:17.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
85572025-04-30T06:43:17.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "649512b6-8520-4320-9d23-061eecf7aa6b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85582025-04-30T06:43:17.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
85592025-04-30T06:43:17.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d74ade-72d8-4ba6-a3aa-1453f7ed167e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85602025-04-30T06:43:17.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
85612025-04-30T06:43:17.859ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
85622025-04-30T06:43:17.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48a5cba-f5ce-4beb-a2d0-cd0105446641", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85632025-04-30T06:43:17.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
85642025-04-30T06:43:17.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74aa238-84f7-4d83-8fc0-c8346a426f76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85652025-04-30T06:43:17.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
85662025-04-30T06:43:17.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fed6316-38a0-48f1-a5b2-76de676597d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85672025-04-30T06:43:17.873ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
85682025-04-30T06:43:17.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed58bc9-9527-4c55-ae3a-6da71eadd8d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85692025-04-30T06:43:17.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
85702025-04-30T06:43:17.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4574608-30d8-41d7-9b88-2d06550da796", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85712025-04-30T06:43:17.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
85722025-04-30T06:43:17.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3059b5db-431d-4c98-92c2-b9c8dc39af18", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
85732025-04-30T06:43:17.876ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85742025-04-30T06:43:17.877ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85752025-04-30T06:43:17.877ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85762025-04-30T06:43:17.877ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85772025-04-30T06:43:17.878ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85782025-04-30T06:43:17.881ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85792025-04-30T06:43:17.889ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85802025-04-30T06:43:17.889ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85812025-04-30T06:43:17.889ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85822025-04-30T06:43:17.889ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85832025-04-30T06:43:17.889ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85842025-04-30T06:43:17.889ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85852025-04-30T06:43:17.892ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85862025-04-30T06:43:17.895ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85872025-04-30T06:43:17.895ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85882025-04-30T06:43:17.896ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85892025-04-30T06:43:17.896ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85902025-04-30T06:43:17.960ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
85912025-04-30T06:43:17.960ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
85922025-04-30T06:43:17.961ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85932025-04-30T06:43:17.961ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85942025-04-30T06:43:17.961ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85952025-04-30T06:43:17.961ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
85962025-04-30T06:43:17.962ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85972025-04-30T06:43:17.962ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:62204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
85982025-04-30T06:43:17.963ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:62204 req_id = 060a10d6-2282-4309-ba46-e0c219f89e24 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85992025-04-30T06:43:17.963ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 475 local_addr = [::1]:45714 method = GET remote_addr = [::1]:62204 req_id = 060a10d6-2282-4309-ba46-e0c219f89e24 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86002025-04-30T06:43:17.963ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "060a10d6-2282-4309-ba46-e0c219f89e24", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86012025-04-30T06:43:17.966ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86022025-04-30T06:43:17.966ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86032025-04-30T06:43:17.966ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
86042025-04-30T06:43:17.969ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86052025-04-30T06:43:17.969ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86062025-04-30T06:43:17.969ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86072025-04-30T06:43:17.969ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86082025-04-30T06:43:17.969ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86092025-04-30T06:43:17.970ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86102025-04-30T06:43:17.977ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86112025-04-30T06:43:17.977ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86122025-04-30T06:43:17.977ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
86132025-04-30T06:43:17.978ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86142025-04-30T06:43:17.979ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86152025-04-30T06:43:17.979ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86162025-04-30T06:43:17.979ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
86172025-04-30T06:43:17.981ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
86182025-04-30T06:43:17.981ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86192025-04-30T06:43:17.985ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86202025-04-30T06:43:17.985ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86212025-04-30T06:43:17.986ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86222025-04-30T06:43:17.986ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86232025-04-30T06:43:17.989ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86242025-04-30T06:43:17.989ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
86252025-04-30T06:43:17.991ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86262025-04-30T06:43:17.993ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
86272025-04-30T06:43:17.993ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
86282025-04-30T06:43:18.000ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
86292025-04-30T06:43:18.001ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86302025-04-30T06:43:18.026ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86312025-04-30T06:43:18.028ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86322025-04-30T06:43:18.028ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 430.861343ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86332025-04-30T06:43:18.028ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86342025-04-30T06:43:18.028ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86352025-04-30T06:43:18.028ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86362025-04-30T06:43:18.028ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86372025-04-30T06:43:18.028ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86382025-04-30T06:43:18.029ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86392025-04-30T06:43:18.033ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86402025-04-30T06:43:18.041ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86412025-04-30T06:43:18.041ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86422025-04-30T06:43:18.041ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86432025-04-30T06:43:18.044ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86442025-04-30T06:43:18.047ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
86452025-04-30T06:43:18.049ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
86462025-04-30T06:43:18.050ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86472025-04-30T06:43:18.052ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 65 05 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 e............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55416 req_id = dc5f7fa3-e2df-43a7-a608-ba710051b6a6
86482025-04-30T06:43:18.052ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25861,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55416 req_id = dc5f7fa3-e2df-43a7-a608-ba710051b6a6
86492025-04-30T06:43:18.052ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
86502025-04-30T06:43:18.052ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86512025-04-30T06:43:18.053ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55416 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dc5f7fa3-e2df-43a7-a608-ba710051b6a6
86522025-04-30T06:43:18.053ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4262, tv_nsec: 852366130 } })
86532025-04-30T06:43:18.053ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
86542025-04-30T06:43:18.053ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .t..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58681 req_id = 8341ec9b-95df-47f4-aa63-77d2e27e6baa
86552025-04-30T06:43:18.053ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33908,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58681 req_id = 8341ec9b-95df-47f4-aa63-77d2e27e6baa
86562025-04-30T06:43:18.053ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86572025-04-30T06:43:18.054ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58681 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8341ec9b-95df-47f4-aa63-77d2e27e6baa
86582025-04-30T06:43:18.112ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86592025-04-30T06:43:18.113ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86602025-04-30T06:43:18.113ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86612025-04-30T06:43:18.113ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86622025-04-30T06:43:18.113ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
86632025-04-30T06:43:18.114ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86642025-04-30T06:43:18.114ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86652025-04-30T06:43:18.115ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 77 60 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c w`..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54299 req_id = fdbf1f17-6c33-4810-b228-1310c4e0f459
86662025-04-30T06:43:18.115ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30560,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54299 req_id = fdbf1f17-6c33-4810-b228-1310c4e0f459
86672025-04-30T06:43:18.116ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
86682025-04-30T06:43:18.116ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30560, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54299 req_id = fdbf1f17-6c33-4810-b228-1310c4e0f459
86692025-04-30T06:43:18.116ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
86702025-04-30T06:43:18.116ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
86712025-04-30T06:43:18.116ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
86722025-04-30T06:43:18.118ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86732025-04-30T06:43:18.123ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86742025-04-30T06:43:18.123ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86752025-04-30T06:43:18.123ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
86762025-04-30T06:43:18.124ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86772025-04-30T06:43:18.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfe8b8e-3033-4095-8393-7e64132e1599", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86782025-04-30T06:43:18.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
86792025-04-30T06:43:18.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7698690e-96de-461d-ad44-7a63295f86d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86802025-04-30T06:43:18.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
86812025-04-30T06:43:18.131ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86822025-04-30T06:43:18.131ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
86832025-04-30T06:43:18.131ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86842025-04-30T06:43:18.131ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "111e4122-648f-486f-80a9-db15b9b9014b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86852025-04-30T06:43:18.131ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
86862025-04-30T06:43:18.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5ec87e-1aa6-4a0e-b5de-2bd73fecb9ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86872025-04-30T06:43:18.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
86882025-04-30T06:43:18.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e806ae9-d5a0-4e79-a97a-fee33023c2d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86892025-04-30T06:43:18.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
86902025-04-30T06:43:18.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "595a5036-bbc5-4efd-ad48-0f3ebc0bde0b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86912025-04-30T06:43:18.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
86922025-04-30T06:43:18.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1a8dd59-36c5-44f2-abb9-ba1cdab2cf13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86932025-04-30T06:43:18.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
86942025-04-30T06:43:18.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a11a3a5-ed80-408f-9ff0-bc13bf3a5cd0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86952025-04-30T06:43:18.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
86962025-04-30T06:43:18.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48b9797-4be5-42bc-bb67-8df4ab7d8660", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86972025-04-30T06:43:18.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
86982025-04-30T06:43:18.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "224e70c6-ae4e-418e-bace-b45076a867c4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
86992025-04-30T06:43:18.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
87002025-04-30T06:43:18.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "734489c5-542f-46aa-ac8d-dd4c19f64c8c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87012025-04-30T06:43:18.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
87022025-04-30T06:43:18.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2a28833-5ad5-4892-b6f8-eb97b51112a7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87032025-04-30T06:43:18.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
87042025-04-30T06:43:18.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff43a79a-47a4-4124-802b-68cdbb4636a1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87052025-04-30T06:43:18.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
87062025-04-30T06:43:18.169ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
87072025-04-30T06:43:18.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c05a1fc4-de83-4c9b-9d91-4f08ddad56be", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87082025-04-30T06:43:18.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
87092025-04-30T06:43:18.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "564fe2f2-8db0-4313-a373-65dff610640d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87102025-04-30T06:43:18.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
87112025-04-30T06:43:18.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5a97f7-317e-4d74-931f-d7dab5cb4050", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:17 GMT"} })
87122025-04-30T06:43:18.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
87132025-04-30T06:43:18.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb9e239a-103b-43da-8f0a-800c421d0645", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87142025-04-30T06:43:18.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
87152025-04-30T06:43:18.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "286ac924-a0a3-4572-8b06-207a8d31e8de", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87162025-04-30T06:43:18.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
87172025-04-30T06:43:18.225ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87182025-04-30T06:43:18.225ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87192025-04-30T06:43:18.225ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87202025-04-30T06:43:18.226ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87212025-04-30T06:43:18.227ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87222025-04-30T06:43:18.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6b5774c-c582-4262-8e51-3122ba42593f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87232025-04-30T06:43:18.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
87242025-04-30T06:43:18.231ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87252025-04-30T06:43:18.241ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87262025-04-30T06:43:18.243ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87272025-04-30T06:43:18.248ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87282025-04-30T06:43:18.249ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87292025-04-30T06:43:18.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d970ae44-0cf0-4f3b-b2f6-f02309514535", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87302025-04-30T06:43:18.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
87312025-04-30T06:43:18.252ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7a87d6d-838c-4d25-bcea-fcf77ec33469", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87322025-04-30T06:43:18.252ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
87332025-04-30T06:43:18.253ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87342025-04-30T06:43:18.253ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87352025-04-30T06:43:18.253ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87362025-04-30T06:43:18.272ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c30d0de-784e-4438-8c01-4be026ed4053", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87372025-04-30T06:43:18.272ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
87382025-04-30T06:43:18.281ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e92e00bf-34b9-4305-ad08-6d6e868b3af0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87392025-04-30T06:43:18.282ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
87402025-04-30T06:43:18.300ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3198b74-a101-4094-9544-d85e9ccb1178", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87412025-04-30T06:43:18.300ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
87422025-04-30T06:43:18.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2934d4a2-cf51-4f79-91c9-ff10a61f5835", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87432025-04-30T06:43:18.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
87442025-04-30T06:43:18.309ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
87452025-04-30T06:43:18.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca219c8-ae0d-4572-967e-75c70c8169d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87462025-04-30T06:43:18.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
87472025-04-30T06:43:18.314ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f7c2680-bc19-4c4b-99f6-3ebdc085ad0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87482025-04-30T06:43:18.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
87492025-04-30T06:43:18.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "197851c7-b5c5-4e31-9bbc-129f7841d338", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87502025-04-30T06:43:18.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
87512025-04-30T06:43:18.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5badb60-425a-4ea8-bb34-fb1e38077155", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87522025-04-30T06:43:18.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
87532025-04-30T06:43:18.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd875a4-17f1-41b8-8925-5d49a6fed28c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87542025-04-30T06:43:18.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
87552025-04-30T06:43:18.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d88ae38e-1477-43f0-972e-4f0f911c2de6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87562025-04-30T06:43:18.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
87572025-04-30T06:43:18.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54b8bf71-146f-41f9-aef7-037e6d2edad7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87582025-04-30T06:43:18.332ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
87592025-04-30T06:43:18.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2c28e68-e0c3-486a-adc8-f7d6d0e0bf36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87602025-04-30T06:43:18.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
87612025-04-30T06:43:18.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad598ed5-c131-4e3d-92ea-004545c647eb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87622025-04-30T06:43:18.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
87632025-04-30T06:43:18.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f77faf2-ef38-4986-8ac2-905830fd4055", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87642025-04-30T06:43:18.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
87652025-04-30T06:43:18.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c4619e-c972-44c5-8525-c5b49d1ad4d3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87662025-04-30T06:43:18.356ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
87672025-04-30T06:43:18.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "441e0e78-1a40-44f8-89e6-db9a4a8c2393", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87682025-04-30T06:43:18.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
87692025-04-30T06:43:18.359ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87702025-04-30T06:43:18.359ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87712025-04-30T06:43:18.360ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87722025-04-30T06:43:18.360ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87732025-04-30T06:43:18.362ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87742025-04-30T06:43:18.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55922d6e-1df7-4181-b846-10603b13cc75", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87752025-04-30T06:43:18.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
87762025-04-30T06:43:18.365ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87772025-04-30T06:43:18.373ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87782025-04-30T06:43:18.373ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87792025-04-30T06:43:18.373ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87802025-04-30T06:43:18.374ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87812025-04-30T06:43:18.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "524bcb92-5491-4d10-9db1-954c30a356d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87822025-04-30T06:43:18.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
87832025-04-30T06:43:18.377ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9c7dc6-bcad-4255-b15d-af727700a915", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87842025-04-30T06:43:18.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
87852025-04-30T06:43:18.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "151e53d5-84b0-4d49-8ead-6523238dd3e6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87862025-04-30T06:43:18.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
87872025-04-30T06:43:18.382ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87882025-04-30T06:43:18.382ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
87892025-04-30T06:43:18.383ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87902025-04-30T06:43:18.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd66df98-1230-4d59-8f8e-8c3e394bec56", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87912025-04-30T06:43:18.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
87922025-04-30T06:43:18.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d43f21-0b4e-4181-8260-87339ea6de53", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87932025-04-30T06:43:18.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
87942025-04-30T06:43:18.390ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a325ed21-1e38-46b4-9e20-cff691b2cf14", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87952025-04-30T06:43:18.390ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
87962025-04-30T06:43:18.396ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ce52d88-86b4-44a0-9024-c9a41203583c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
87972025-04-30T06:43:18.396ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87982025-04-30T06:43:18.396ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87992025-04-30T06:43:18.396ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88002025-04-30T06:43:18.396ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88012025-04-30T06:43:18.397ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88022025-04-30T06:43:18.400ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88032025-04-30T06:43:18.404ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88042025-04-30T06:43:18.404ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88052025-04-30T06:43:18.404ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88062025-04-30T06:43:18.404ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88072025-04-30T06:43:18.404ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88082025-04-30T06:43:18.404ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88092025-04-30T06:43:18.407ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88102025-04-30T06:43:18.414ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88112025-04-30T06:43:18.414ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88122025-04-30T06:43:18.414ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88132025-04-30T06:43:18.414ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88142025-04-30T06:43:18.464ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
88152025-04-30T06:43:18.464ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
88162025-04-30T06:43:18.466ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:41506 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88172025-04-30T06:43:18.467ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:41506 req_id = 35aba390-f45d-4a69-9903-25e4f3bd7586 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88182025-04-30T06:43:18.467ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 560 local_addr = [::1]:45714 method = GET remote_addr = [::1]:41506 req_id = 35aba390-f45d-4a69-9903-25e4f3bd7586 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88192025-04-30T06:43:18.467ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "35aba390-f45d-4a69-9903-25e4f3bd7586", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88202025-04-30T06:43:18.470ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88212025-04-30T06:43:18.470ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88222025-04-30T06:43:18.470ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
88232025-04-30T06:43:18.472ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88242025-04-30T06:43:18.472ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88252025-04-30T06:43:18.473ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88262025-04-30T06:43:18.473ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88272025-04-30T06:43:18.473ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88282025-04-30T06:43:18.484ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88292025-04-30T06:43:18.488ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88302025-04-30T06:43:18.488ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88312025-04-30T06:43:18.488ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88322025-04-30T06:43:18.488ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88332025-04-30T06:43:18.489ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
88342025-04-30T06:43:18.527ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88352025-04-30T06:43:18.527ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88362025-04-30T06:43:18.527ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 499.640211ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88372025-04-30T06:43:18.527ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88382025-04-30T06:43:18.527ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88392025-04-30T06:43:18.527ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88402025-04-30T06:43:18.528ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88412025-04-30T06:43:18.528ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88422025-04-30T06:43:18.528ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88432025-04-30T06:43:18.530ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88442025-04-30T06:43:18.534ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88452025-04-30T06:43:18.534ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88462025-04-30T06:43:18.534ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88472025-04-30T06:43:18.534ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88482025-04-30T06:43:18.539ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
88492025-04-30T06:43:18.539ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88502025-04-30T06:43:18.540ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d b7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49996 req_id = 324efd13-f9e8-4af5-a65f-22c2a53078b4
88512025-04-30T06:43:18.540ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49996 req_id = 324efd13-f9e8-4af5-a65f-22c2a53078b4
88522025-04-30T06:43:18.540ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
88532025-04-30T06:43:18.540ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88542025-04-30T06:43:18.540ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49996 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 324efd13-f9e8-4af5-a65f-22c2a53078b4
88552025-04-30T06:43:18.542ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88562025-04-30T06:43:18.542ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88572025-04-30T06:43:18.542ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88582025-04-30T06:43:18.542ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
88592025-04-30T06:43:18.544ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88602025-04-30T06:43:18.544ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 343610462 } })
88612025-04-30T06:43:18.544ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
88622025-04-30T06:43:18.546ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: dc fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50715 req_id = dd933366-c50d-4a0e-ba2f-886ae5708d7c
88632025-04-30T06:43:18.546ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56574,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50715 req_id = dd933366-c50d-4a0e-ba2f-886ae5708d7c
88642025-04-30T06:43:18.546ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88652025-04-30T06:43:18.547ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50715 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dd933366-c50d-4a0e-ba2f-886ae5708d7c
88662025-04-30T06:43:18.628ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
88672025-04-30T06:43:18.628ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88682025-04-30T06:43:18.635ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88692025-04-30T06:43:18.635ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88702025-04-30T06:43:18.635ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
88712025-04-30T06:43:18.636ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88722025-04-30T06:43:18.636ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 59 28 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Y(..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54792 req_id = 5ac1665e-588e-4264-b50f-922c1de643f2
88732025-04-30T06:43:18.636ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22824,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54792 req_id = 5ac1665e-588e-4264-b50f-922c1de643f2
88742025-04-30T06:43:18.637ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
88752025-04-30T06:43:18.637ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22824, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54792 req_id = 5ac1665e-588e-4264-b50f-922c1de643f2
88762025-04-30T06:43:18.638ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
88772025-04-30T06:43:18.638ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
88782025-04-30T06:43:18.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
88792025-04-30T06:43:18.639ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88802025-04-30T06:43:18.639ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
88812025-04-30T06:43:18.641ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88822025-04-30T06:43:18.643ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "21eb15b7-77a2-40d8-8983-9ef70624903d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88832025-04-30T06:43:18.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
88842025-04-30T06:43:18.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b219e6-0c63-46ee-991a-143e8a712781", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88852025-04-30T06:43:18.650ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
88862025-04-30T06:43:18.655ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aae5f619-bba2-4a05-bb27-5a19e9f18dff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88872025-04-30T06:43:18.655ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
88882025-04-30T06:43:18.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b4603f8-830e-416b-9605-c5bd78009280", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88892025-04-30T06:43:18.662ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
88902025-04-30T06:43:18.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b560d5ae-3392-4b83-b92d-b5473e9276cc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88912025-04-30T06:43:18.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
88922025-04-30T06:43:18.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d38ff2-2cd1-49d7-9e50-4e2da6566ca3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88932025-04-30T06:43:18.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
88942025-04-30T06:43:18.676ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
88952025-04-30T06:43:18.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b685bebe-9a98-4b19-8c56-fe30db0c6a8e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88962025-04-30T06:43:18.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
88972025-04-30T06:43:18.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e743d153-a0e3-420c-b876-8b7c0fc2ae98", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
88982025-04-30T06:43:18.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
88992025-04-30T06:43:18.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a220b5c-5003-4d22-a407-073ad576ccda", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89002025-04-30T06:43:18.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
89012025-04-30T06:43:18.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08684c4a-2a02-47e9-b2ec-454104b53f1c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89022025-04-30T06:43:18.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
89032025-04-30T06:43:18.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e55342-6e0b-413f-a801-f5fe7d44a981", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89042025-04-30T06:43:18.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
89052025-04-30T06:43:18.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af6508f0-0cc7-4887-a567-6b9e770c714e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89062025-04-30T06:43:18.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
89072025-04-30T06:43:18.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "851442f3-980a-41f1-a669-5d3488fa93b2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89082025-04-30T06:43:18.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
89092025-04-30T06:43:18.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "216b0157-3810-4dcd-8a3f-9c05b24f04ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89102025-04-30T06:43:18.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
89112025-04-30T06:43:18.695ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e6176ea-4740-41ca-a63b-56308a48457a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89122025-04-30T06:43:18.695ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
89132025-04-30T06:43:18.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "179d846f-072f-4ef0-b453-4d6f5f9848d3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89142025-04-30T06:43:18.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
89152025-04-30T06:43:18.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2bc8ef4-354c-488f-9b3b-8dfe803d66c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89162025-04-30T06:43:18.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
89172025-04-30T06:43:18.704ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89182025-04-30T06:43:18.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89192025-04-30T06:43:18.706ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89202025-04-30T06:43:18.707ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89212025-04-30T06:43:18.707ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89222025-04-30T06:43:18.707ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89232025-04-30T06:43:18.708ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c5d526-48eb-4371-859f-5cb1dbb64004", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89242025-04-30T06:43:18.708ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
89252025-04-30T06:43:18.711ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a78e4ef-89a7-42b5-89eb-ee403c8afa9b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89262025-04-30T06:43:18.711ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
89272025-04-30T06:43:18.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7853b580-90ac-48fc-97a4-810f995902f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89282025-04-30T06:43:18.713ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
89292025-04-30T06:43:18.714ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e1746a-6f47-44dc-b4b9-7fa498df6b7e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89302025-04-30T06:43:18.715ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
89312025-04-30T06:43:18.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c01bd78-2b9e-4511-990b-6d25edda61a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89322025-04-30T06:43:18.717ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
89332025-04-30T06:43:18.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9850336-7242-4fbe-a50c-123176918f62", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89342025-04-30T06:43:18.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
89352025-04-30T06:43:18.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6b9a90-84c6-4e34-bd91-d85bcf869861", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89362025-04-30T06:43:18.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
89372025-04-30T06:43:18.722ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "923c5c84-4534-4ed3-b796-f5924124fbe3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89382025-04-30T06:43:18.722ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
89392025-04-30T06:43:18.725ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30e6bc42-bd70-400d-85e9-f6e070cca92b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89402025-04-30T06:43:18.725ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
89412025-04-30T06:43:18.726ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89422025-04-30T06:43:18.726ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89432025-04-30T06:43:18.727ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89442025-04-30T06:43:18.727ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
89452025-04-30T06:43:18.727ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89462025-04-30T06:43:18.728ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72c837bb-ace7-4308-8969-0e26dd2e3ec8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89472025-04-30T06:43:18.728ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
89482025-04-30T06:43:18.730ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89492025-04-30T06:43:18.735ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89502025-04-30T06:43:18.735ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89512025-04-30T06:43:18.735ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
89522025-04-30T06:43:18.736ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89532025-04-30T06:43:18.738ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5c6cb7-162d-4d4f-959e-ba2af54307a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89542025-04-30T06:43:18.738ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
89552025-04-30T06:43:18.739ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89562025-04-30T06:43:18.739ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
89572025-04-30T06:43:18.739ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89582025-04-30T06:43:18.745ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9478e692-5fb5-474c-850b-ac6aa5458fe4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89592025-04-30T06:43:18.745ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
89602025-04-30T06:43:18.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5784e7-a444-4bc3-b4de-655e35d52ae8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89612025-04-30T06:43:18.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
89622025-04-30T06:43:18.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9920a0a9-cab0-41c2-8920-7b0f180ba2e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89632025-04-30T06:43:18.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
89642025-04-30T06:43:18.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e020d049-668f-4c50-ad05-4aa34c902f18", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89652025-04-30T06:43:18.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
89662025-04-30T06:43:18.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d8dd65-1c4a-4bbb-bb47-4b3cb6ded257", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89672025-04-30T06:43:18.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
89682025-04-30T06:43:18.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aaf0777-d479-44b0-a3d0-d4ff14a29e5c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89692025-04-30T06:43:18.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
89702025-04-30T06:43:18.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a218c754-af10-4d6b-b140-5f0cc7c27f64", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89712025-04-30T06:43:18.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
89722025-04-30T06:43:18.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6572976e-7203-4003-ac30-27c1fc92ec48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89732025-04-30T06:43:18.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
89742025-04-30T06:43:18.779ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
89752025-04-30T06:43:18.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74a8111-8521-43a4-bd85-8cdbdeda92ce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89762025-04-30T06:43:18.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
89772025-04-30T06:43:18.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48f4a2b6-68a4-4a0e-b9d1-be47e382cd46", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89782025-04-30T06:43:18.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
89792025-04-30T06:43:18.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76ea65f5-416d-4dfe-88b3-2e122d883c8a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89802025-04-30T06:43:18.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
89812025-04-30T06:43:18.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a48a734-2a11-4630-abb3-936df09b12ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89822025-04-30T06:43:18.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
89832025-04-30T06:43:18.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d8f3510-313d-433d-9b10-158169ebbf02", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89842025-04-30T06:43:18.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
89852025-04-30T06:43:18.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f98ac716-d7d5-428e-9ad8-30e0f91871e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89862025-04-30T06:43:18.800ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
89872025-04-30T06:43:18.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc062671-eb41-41ab-818a-6500326c85a0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89882025-04-30T06:43:18.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
89892025-04-30T06:43:18.804ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e18e61-fb1d-4edf-961a-2281c3c56ecb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89902025-04-30T06:43:18.804ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
89912025-04-30T06:43:18.806ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80eac3e-5154-4d64-a0fc-03c0200d32df", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
89922025-04-30T06:43:18.806ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89932025-04-30T06:43:18.806ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89942025-04-30T06:43:18.806ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89952025-04-30T06:43:18.807ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89962025-04-30T06:43:18.808ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89972025-04-30T06:43:18.812ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89982025-04-30T06:43:18.816ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89992025-04-30T06:43:18.816ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90002025-04-30T06:43:18.816ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90012025-04-30T06:43:18.816ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90022025-04-30T06:43:18.816ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90032025-04-30T06:43:18.818ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90042025-04-30T06:43:18.820ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90052025-04-30T06:43:18.827ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90062025-04-30T06:43:18.828ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90072025-04-30T06:43:18.828ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90082025-04-30T06:43:18.830ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90092025-04-30T06:43:18.831ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90102025-04-30T06:43:18.831ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90112025-04-30T06:43:18.831ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90122025-04-30T06:43:18.831ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
90132025-04-30T06:43:18.832ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90142025-04-30T06:43:18.834ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90152025-04-30T06:43:18.846ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90162025-04-30T06:43:18.847ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90172025-04-30T06:43:18.847ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
90182025-04-30T06:43:18.847ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90192025-04-30T06:43:18.920ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
90202025-04-30T06:43:18.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
90212025-04-30T06:43:18.921ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:50559 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90222025-04-30T06:43:18.922ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:50559 req_id = f970fbe1-fab4-4f13-b5fb-d1f9429858c9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90232025-04-30T06:43:18.923ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 998 local_addr = [::1]:45714 method = GET remote_addr = [::1]:50559 req_id = f970fbe1-fab4-4f13-b5fb-d1f9429858c9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90242025-04-30T06:43:18.923ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90252025-04-30T06:43:18.923ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
90262025-04-30T06:43:18.924ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90272025-04-30T06:43:18.924ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f970fbe1-fab4-4f13-b5fb-d1f9429858c9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
90282025-04-30T06:43:18.927ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90292025-04-30T06:43:18.927ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90302025-04-30T06:43:18.927ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
90312025-04-30T06:43:18.930ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90322025-04-30T06:43:18.930ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90332025-04-30T06:43:18.930ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90342025-04-30T06:43:18.930ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90352025-04-30T06:43:18.930ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90362025-04-30T06:43:18.936ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90372025-04-30T06:43:18.945ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90382025-04-30T06:43:18.946ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90392025-04-30T06:43:18.946ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90402025-04-30T06:43:18.947ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90412025-04-30T06:43:18.975ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90422025-04-30T06:43:18.975ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90432025-04-30T06:43:18.976ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
90442025-04-30T06:43:18.978ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
90452025-04-30T06:43:18.984ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
90462025-04-30T06:43:18.985ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
90472025-04-30T06:43:18.990ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
90482025-04-30T06:43:18.990ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90492025-04-30T06:43:18.997ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90502025-04-30T06:43:19.001ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90512025-04-30T06:43:19.001ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 473.31583ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90522025-04-30T06:43:19.001ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90532025-04-30T06:43:19.001ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90542025-04-30T06:43:19.001ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90552025-04-30T06:43:19.001ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90562025-04-30T06:43:19.001ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90572025-04-30T06:43:19.002ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90582025-04-30T06:43:19.004ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
90592025-04-30T06:43:19.015ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90602025-04-30T06:43:19.023ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90612025-04-30T06:43:19.023ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90622025-04-30T06:43:19.024ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90632025-04-30T06:43:19.024ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90642025-04-30T06:43:19.029ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
90652025-04-30T06:43:19.030ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
90662025-04-30T06:43:19.030ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bb 33 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .3..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61066 req_id = 7b0d6c7f-01fb-424e-b9a0-a615242817bf
90672025-04-30T06:43:19.031ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47923,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61066 req_id = 7b0d6c7f-01fb-424e-b9a0-a615242817bf
90682025-04-30T06:43:19.031ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
90692025-04-30T06:43:19.031ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90702025-04-30T06:43:19.031ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61066 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7b0d6c7f-01fb-424e-b9a0-a615242817bf
90712025-04-30T06:43:19.031ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 830831278 } })
90722025-04-30T06:43:19.031ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
90732025-04-30T06:43:19.032ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f0 8f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55759 req_id = 390ff1cb-5f7b-4b8e-8709-141619532f1f
90742025-04-30T06:43:19.032ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61583,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55759 req_id = 390ff1cb-5f7b-4b8e-8709-141619532f1f
90752025-04-30T06:43:19.032ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90762025-04-30T06:43:19.032ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55759 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 390ff1cb-5f7b-4b8e-8709-141619532f1f
90772025-04-30T06:43:19.087ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
90782025-04-30T06:43:19.087ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90792025-04-30T06:43:19.087ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90802025-04-30T06:43:19.087ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90812025-04-30T06:43:19.087ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
90822025-04-30T06:43:19.088ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90832025-04-30T06:43:19.088ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90842025-04-30T06:43:19.089ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c0 2c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .,..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53562 req_id = a65d03df-cc75-4f01-a0a2-735ef948d1a1
90852025-04-30T06:43:19.089ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49196,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53562 req_id = a65d03df-cc75-4f01-a0a2-735ef948d1a1
90862025-04-30T06:43:19.089ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
90872025-04-30T06:43:19.089ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49196, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53562 req_id = a65d03df-cc75-4f01-a0a2-735ef948d1a1
90882025-04-30T06:43:19.091ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
90892025-04-30T06:43:19.091ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
90902025-04-30T06:43:19.091ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
90912025-04-30T06:43:19.091ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90922025-04-30T06:43:19.096ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90932025-04-30T06:43:19.096ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90942025-04-30T06:43:19.096ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
90952025-04-30T06:43:19.097ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90962025-04-30T06:43:19.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9774ff6c-f240-4196-b0ec-13c424a3c820", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
90972025-04-30T06:43:19.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
90982025-04-30T06:43:19.099ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90992025-04-30T06:43:19.099ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
91002025-04-30T06:43:19.100ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91012025-04-30T06:43:19.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c1471ec-bdd0-4a9d-ada6-93954bb1d51d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91022025-04-30T06:43:19.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
91032025-04-30T06:43:19.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ed61c74-db5d-41c3-9be0-81018d553e5b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91042025-04-30T06:43:19.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
91052025-04-30T06:43:19.114ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51bc428b-fade-42bc-b8bb-3cf54598cb43", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91062025-04-30T06:43:19.114ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
91072025-04-30T06:43:19.118ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcf7181f-0bde-4fc7-8d05-a1a64ee9fa3d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91082025-04-30T06:43:19.118ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
91092025-04-30T06:43:19.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d4aec84-773d-4e02-b42a-576124d680ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91102025-04-30T06:43:19.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
91112025-04-30T06:43:19.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b727fda0-2e80-4e13-aa33-c807fd0327b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91122025-04-30T06:43:19.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
91132025-04-30T06:43:19.130ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
91142025-04-30T06:43:19.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8db8d09-9f4e-44c2-907c-509e6c6b015a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91152025-04-30T06:43:19.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
91162025-04-30T06:43:19.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a26a4de-d699-42cd-8a4c-93c1f526a14c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91172025-04-30T06:43:19.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
91182025-04-30T06:43:19.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "524e083d-dc4a-4fb6-b7ea-19b923ff330c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91192025-04-30T06:43:19.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
91202025-04-30T06:43:19.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e09c2f-8ba9-48da-aebc-6410cd1632ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91212025-04-30T06:43:19.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
91222025-04-30T06:43:19.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3619dd34-dfe6-4d13-83ce-2dff8c074e26", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91232025-04-30T06:43:19.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
91242025-04-30T06:43:19.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b14cecff-18af-4e16-89a3-a6bf7217fe77", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91252025-04-30T06:43:19.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
91262025-04-30T06:43:19.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7262c2d-f7bc-4db9-ad35-2307a965ff45", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91272025-04-30T06:43:19.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
91282025-04-30T06:43:19.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d4a565-f125-4325-a413-6acac533f7d4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91292025-04-30T06:43:19.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
91302025-04-30T06:43:19.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dbcfdff-2548-4ec6-b087-a0d019448baa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91312025-04-30T06:43:19.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
91322025-04-30T06:43:19.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57f7dfac-271a-4380-9026-1977ff225cff", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91332025-04-30T06:43:19.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
91342025-04-30T06:43:19.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae81704b-71f6-4c99-88cb-435950054b1a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91352025-04-30T06:43:19.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
91362025-04-30T06:43:19.165ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "439ebcba-ba10-45fc-a195-5c330c9920b7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91372025-04-30T06:43:19.165ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
91382025-04-30T06:43:19.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ef0db2-762e-494a-8831-e1a58bb9c4da", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91392025-04-30T06:43:19.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
91402025-04-30T06:43:19.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f902e21-3a6b-46f2-9094-19862e29cf8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91412025-04-30T06:43:19.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
91422025-04-30T06:43:19.170ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc6fe18e-aa49-4074-ae5a-dd294e63ae9f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91432025-04-30T06:43:19.170ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
91442025-04-30T06:43:19.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "558d7b64-3936-467f-af39-2885389341c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91452025-04-30T06:43:19.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
91462025-04-30T06:43:19.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16226581-beeb-4382-a86b-d113d977f979", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91472025-04-30T06:43:19.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
91482025-04-30T06:43:19.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33989ece-49c9-4c9d-8b31-7f4ec40f828d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91492025-04-30T06:43:19.177ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
91502025-04-30T06:43:19.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "423f937d-ec15-4c8e-9a59-3cc738703775", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:18 GMT"} })
91512025-04-30T06:43:19.179ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
91522025-04-30T06:43:19.180ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91532025-04-30T06:43:19.180ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91542025-04-30T06:43:19.181ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91552025-04-30T06:43:19.181ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
91562025-04-30T06:43:19.181ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91572025-04-30T06:43:19.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73e244c2-7c93-416d-8cdc-78d2da21e580", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91582025-04-30T06:43:19.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
91592025-04-30T06:43:19.184ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91602025-04-30T06:43:19.192ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91612025-04-30T06:43:19.192ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91622025-04-30T06:43:19.192ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
91632025-04-30T06:43:19.192ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91642025-04-30T06:43:19.193ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "badb6ed5-b4f7-45bb-b1b9-4ecea487c5a4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91652025-04-30T06:43:19.193ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
91662025-04-30T06:43:19.196ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91672025-04-30T06:43:19.196ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
91682025-04-30T06:43:19.197ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91692025-04-30T06:43:19.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4341061c-dfa9-4acb-81c1-63eb4a67b318", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91702025-04-30T06:43:19.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
91712025-04-30T06:43:19.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2047509-0232-4de0-901a-4ee04dcd41de", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91722025-04-30T06:43:19.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
91732025-04-30T06:43:19.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de5abc44-929e-46b4-ba63-6223a7fe5c98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91742025-04-30T06:43:19.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
91752025-04-30T06:43:19.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0038ec9-3f55-4b99-b971-1397a3cce07a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91762025-04-30T06:43:19.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
91772025-04-30T06:43:19.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56d4a96c-9bd4-42ed-9f67-bf9a3e75fdb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91782025-04-30T06:43:19.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
91792025-04-30T06:43:19.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3d07f7d-e8bf-4dbd-b02c-b4df48a22764", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91802025-04-30T06:43:19.222ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
91812025-04-30T06:43:19.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0952b5c1-e139-4468-a9c9-0da8a1e9d857", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91822025-04-30T06:43:19.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
91832025-04-30T06:43:19.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c12de756-98b5-4688-ab76-34ec5bcc588a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91842025-04-30T06:43:19.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
91852025-04-30T06:43:19.232ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
91862025-04-30T06:43:19.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29af16d2-8938-431a-8a99-59196d63df78", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91872025-04-30T06:43:19.233ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
91882025-04-30T06:43:19.236ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8909bd12-9bb8-45bb-8e8d-753527b3760f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91892025-04-30T06:43:19.237ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
91902025-04-30T06:43:19.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "901a0c94-c146-40e3-b21e-2b9671c295b7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91912025-04-30T06:43:19.239ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
91922025-04-30T06:43:19.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5be33bc3-987e-4a5f-b43c-2fd9c35042c3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91932025-04-30T06:43:19.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
91942025-04-30T06:43:19.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b3c370a-9537-4148-93cb-feb44e6e19f5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91952025-04-30T06:43:19.244ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
91962025-04-30T06:43:19.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "989268d7-1ee3-49d0-92b5-1966ed191ab3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91972025-04-30T06:43:19.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
91982025-04-30T06:43:19.256ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1879bf6c-cecb-4915-aeef-1557a690ddec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
91992025-04-30T06:43:19.256ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
92002025-04-30T06:43:19.271ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d8a94ae-e149-46fd-a839-529f4e9622a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
92012025-04-30T06:43:19.271ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
92022025-04-30T06:43:19.283ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11400c2f-9e2b-4328-aec5-165f341f8b3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
92032025-04-30T06:43:19.283ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92042025-04-30T06:43:19.283ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92052025-04-30T06:43:19.283ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92062025-04-30T06:43:19.283ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92072025-04-30T06:43:19.287ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92082025-04-30T06:43:19.289ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92092025-04-30T06:43:19.289ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92102025-04-30T06:43:19.289ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92112025-04-30T06:43:19.289ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
92122025-04-30T06:43:19.290ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92132025-04-30T06:43:19.295ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92142025-04-30T06:43:19.299ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92152025-04-30T06:43:19.299ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92162025-04-30T06:43:19.299ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92172025-04-30T06:43:19.299ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92182025-04-30T06:43:19.299ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92192025-04-30T06:43:19.300ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92202025-04-30T06:43:19.303ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92212025-04-30T06:43:19.308ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92222025-04-30T06:43:19.308ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92232025-04-30T06:43:19.308ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
92242025-04-30T06:43:19.309ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92252025-04-30T06:43:19.309ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92262025-04-30T06:43:19.313ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92272025-04-30T06:43:19.313ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92282025-04-30T06:43:19.313ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92292025-04-30T06:43:19.314ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92302025-04-30T06:43:19.319ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92312025-04-30T06:43:19.319ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
92322025-04-30T06:43:19.319ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92332025-04-30T06:43:19.381ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
92342025-04-30T06:43:19.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
92352025-04-30T06:43:19.383ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:63425 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92362025-04-30T06:43:19.385ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:63425 req_id = a71d4f6c-0156-4bd5-b676-f49cd729bd72 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92372025-04-30T06:43:19.385ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 613 local_addr = [::1]:45714 method = GET remote_addr = [::1]:63425 req_id = a71d4f6c-0156-4bd5-b676-f49cd729bd72 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92382025-04-30T06:43:19.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a71d4f6c-0156-4bd5-b676-f49cd729bd72", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
92392025-04-30T06:43:19.388ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92402025-04-30T06:43:19.388ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92412025-04-30T06:43:19.388ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
92422025-04-30T06:43:19.389ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92432025-04-30T06:43:19.389ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92442025-04-30T06:43:19.390ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92452025-04-30T06:43:19.390ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92462025-04-30T06:43:19.390ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92472025-04-30T06:43:19.406ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92482025-04-30T06:43:19.412ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92492025-04-30T06:43:19.412ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92502025-04-30T06:43:19.412ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92512025-04-30T06:43:19.412ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92522025-04-30T06:43:19.475ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92532025-04-30T06:43:19.475ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92542025-04-30T06:43:19.475ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 474.100695ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92552025-04-30T06:43:19.475ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
92562025-04-30T06:43:19.475ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92572025-04-30T06:43:19.475ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92582025-04-30T06:43:19.475ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92592025-04-30T06:43:19.475ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92602025-04-30T06:43:19.477ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92612025-04-30T06:43:19.477ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
92622025-04-30T06:43:19.481ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92632025-04-30T06:43:19.486ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92642025-04-30T06:43:19.486ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92652025-04-30T06:43:19.487ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92662025-04-30T06:43:19.487ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92672025-04-30T06:43:19.490ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
92682025-04-30T06:43:19.490ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
92692025-04-30T06:43:19.491ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8a e8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63521 req_id = bc38f414-9dd2-4c27-9d0c-06b62ede8f78
92702025-04-30T06:43:19.491ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35560,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63521 req_id = bc38f414-9dd2-4c27-9d0c-06b62ede8f78
92712025-04-30T06:43:19.492ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
92722025-04-30T06:43:19.492ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92732025-04-30T06:43:19.492ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63521 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bc38f414-9dd2-4c27-9d0c-06b62ede8f78
92742025-04-30T06:43:19.493ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 292089805 } })
92752025-04-30T06:43:19.493ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
92762025-04-30T06:43:19.493ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4f 79 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Oy..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53149 req_id = e7efc33b-d929-4fc6-90c0-5245c0eacd1a
92772025-04-30T06:43:19.493ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20345,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53149 req_id = e7efc33b-d929-4fc6-90c0-5245c0eacd1a
92782025-04-30T06:43:19.493ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92792025-04-30T06:43:19.494ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53149 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e7efc33b-d929-4fc6-90c0-5245c0eacd1a
92802025-04-30T06:43:19.543ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
92812025-04-30T06:43:19.543ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92822025-04-30T06:43:19.544ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92832025-04-30T06:43:19.544ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92842025-04-30T06:43:19.544ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
92852025-04-30T06:43:19.545ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92862025-04-30T06:43:19.545ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 02 54 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .T..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56564 req_id = 325ae049-19a8-4571-b3c4-0f89c6b2489f
92872025-04-30T06:43:19.545ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 596,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56564 req_id = 325ae049-19a8-4571-b3c4-0f89c6b2489f
92882025-04-30T06:43:19.545ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
92892025-04-30T06:43:19.545ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 596, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56564 req_id = 325ae049-19a8-4571-b3c4-0f89c6b2489f
92902025-04-30T06:43:19.546ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
92912025-04-30T06:43:19.546ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
92922025-04-30T06:43:19.546ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
92932025-04-30T06:43:19.548ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1118257a-8631-4751-8e23-8d2ace674706", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
92942025-04-30T06:43:19.548ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
92952025-04-30T06:43:19.549ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92962025-04-30T06:43:19.554ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92972025-04-30T06:43:19.554ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92982025-04-30T06:43:19.554ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
92992025-04-30T06:43:19.554ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93002025-04-30T06:43:19.559ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93012025-04-30T06:43:19.559ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
93022025-04-30T06:43:19.560ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93032025-04-30T06:43:19.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af88d6e-1226-4059-883d-829adc50d7bc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93042025-04-30T06:43:19.562ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
93052025-04-30T06:43:19.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e237bb20-0429-4b8b-bbba-dbd2403cc189", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93062025-04-30T06:43:19.565ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
93072025-04-30T06:43:19.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a76f83d-72e0-45ea-8831-5ca26956e131", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93082025-04-30T06:43:19.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
93092025-04-30T06:43:19.569ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33c5684f-9c1b-4afc-a2fe-0444bb601992", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93102025-04-30T06:43:19.570ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
93112025-04-30T06:43:19.572ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8231907-523a-4522-af38-ecf7e5af2339", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93122025-04-30T06:43:19.572ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
93132025-04-30T06:43:19.574ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52f4c98e-c529-4c35-8c35-85ac816be1a1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93142025-04-30T06:43:19.574ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
93152025-04-30T06:43:19.575ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df6932fe-270a-498c-b2fa-ea1c81e7addf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93162025-04-30T06:43:19.576ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
93172025-04-30T06:43:19.577ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35006ef3-bf0a-4a08-bc88-5b08fbdedcd4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93182025-04-30T06:43:19.577ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
93192025-04-30T06:43:19.578ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f19c55b7-36f6-4b35-811b-d3cc2cc2898b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93202025-04-30T06:43:19.578ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
93212025-04-30T06:43:19.579ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d315f615-2ab5-4ebf-bf9e-324a696abbcd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93222025-04-30T06:43:19.579ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
93232025-04-30T06:43:19.581ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9fefe0-9c3a-4a3b-a5b2-b6b5b0f801f4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93242025-04-30T06:43:19.581ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
93252025-04-30T06:43:19.583ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19c6ba3-fd2d-4cab-9021-0c5ac1520e15", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93262025-04-30T06:43:19.583ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
93272025-04-30T06:43:19.590ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c605b2ad-7bd8-424d-b44e-71e7cede3339", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93282025-04-30T06:43:19.590ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
93292025-04-30T06:43:19.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "614b8a9d-8476-4c3a-b16d-db531ad3cf71", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93302025-04-30T06:43:19.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
93312025-04-30T06:43:19.606ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efff060b-e320-4951-a9c8-75323dfcb3fe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93322025-04-30T06:43:19.607ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
93332025-04-30T06:43:19.616ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfc3a2a2-d501-4e18-87f7-763753eda5da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93342025-04-30T06:43:19.617ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
93352025-04-30T06:43:19.622ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc92aeed-3dae-49aa-b92c-b3549d604e12", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93362025-04-30T06:43:19.622ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
93372025-04-30T06:43:19.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5675ecb-97a7-4452-8525-fabe85ac4a2e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93382025-04-30T06:43:19.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
93392025-04-30T06:43:19.629ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
93402025-04-30T06:43:19.635ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
93412025-04-30T06:43:19.636ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aee40fa-bef0-4e75-9acd-920d05e35407", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93422025-04-30T06:43:19.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
93432025-04-30T06:43:19.639ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93442025-04-30T06:43:19.641ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 1b19bfc5-e8c9-411a-bf46-883db0eba432 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93452025-04-30T06:43:19.643ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 1b19bfc5-e8c9-411a-bf46-883db0eba432 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93462025-04-30T06:43:19.644ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 1b19bfc5-e8c9-411a-bf46-883db0eba432 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93472025-04-30T06:43:19.646ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 1b19bfc5-e8c9-411a-bf46-883db0eba432 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93482025-04-30T06:43:19.648ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe089ee-5442-42be-9c5b-ffcc31f1bfac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93492025-04-30T06:43:19.648ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
93502025-04-30T06:43:19.650ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 9420 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 1b19bfc5-e8c9-411a-bf46-883db0eba432 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93512025-04-30T06:43:19.651ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b19bfc5-e8c9-411a-bf46-883db0eba432", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93522025-04-30T06:43:19.651ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93532025-04-30T06:43:19.652ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 63af08f1-a816-4670-8146-89177e16b395 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93542025-04-30T06:43:19.652ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 63af08f1-a816-4670-8146-89177e16b395 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93552025-04-30T06:43:19.652ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 63af08f1-a816-4670-8146-89177e16b395 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93562025-04-30T06:43:19.653ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 63af08f1-a816-4670-8146-89177e16b395 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93572025-04-30T06:43:19.664ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c0273f9-7add-46fe-9df0-213ed1a60644", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93582025-04-30T06:43:19.664ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
93592025-04-30T06:43:19.670ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 15130 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 63af08f1-a816-4670-8146-89177e16b395 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
93602025-04-30T06:43:19.670ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "63af08f1-a816-4670-8146-89177e16b395", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93612025-04-30T06:43:19.670ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93622025-04-30T06:43:19.670ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
93632025-04-30T06:43:19.670ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
93642025-04-30T06:43:19.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0c93ff9-1bcc-4379-a3b6-7a1dca2c5715", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93652025-04-30T06:43:19.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
93662025-04-30T06:43:19.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab7e888c-a699-4e9f-9b57-a52c86c2ec80", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93672025-04-30T06:43:19.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
93682025-04-30T06:43:19.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b07a02a-d8b9-4ec9-8384-34cccc6b434e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93692025-04-30T06:43:19.673ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
93702025-04-30T06:43:19.676ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f1d707-57d5-4d4a-b011-42df87a7e562", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93712025-04-30T06:43:19.676ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
93722025-04-30T06:43:19.679ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c09b7ce0-7668-4b26-9891-33d2bad8c136", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93732025-04-30T06:43:19.679ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
93742025-04-30T06:43:19.679ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93752025-04-30T06:43:19.680ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93762025-04-30T06:43:19.680ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93772025-04-30T06:43:19.680ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
93782025-04-30T06:43:19.681ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93792025-04-30T06:43:19.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fbfa57-931e-4253-b562-494ef38e8c09", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93802025-04-30T06:43:19.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
93812025-04-30T06:43:19.687ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93822025-04-30T06:43:19.704ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93832025-04-30T06:43:19.704ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93842025-04-30T06:43:19.704ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
93852025-04-30T06:43:19.713ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93862025-04-30T06:43:19.724ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93872025-04-30T06:43:19.724ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93882025-04-30T06:43:19.727ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93892025-04-30T06:43:19.727ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
93902025-04-30T06:43:19.729ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93912025-04-30T06:43:19.730ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93922025-04-30T06:43:19.731ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93932025-04-30T06:43:19.731ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93942025-04-30T06:43:19.731ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93952025-04-30T06:43:19.734ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "034d99cb-e125-4d77-81c6-55bce6c95a8a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93962025-04-30T06:43:19.734ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
93972025-04-30T06:43:19.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "968cc790-4fd4-4773-bc53-f69671bd9dae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
93982025-04-30T06:43:19.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
93992025-04-30T06:43:19.751ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ee238f-00e4-4c7a-8479-62ba5a2c3f73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94002025-04-30T06:43:19.751ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
94012025-04-30T06:43:19.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7340015-4ad5-4cb3-934f-69afb34ac467", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94022025-04-30T06:43:19.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
94032025-04-30T06:43:19.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "827c8a80-9bf6-4a31-a03f-36a0b3a15717", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94042025-04-30T06:43:19.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
94052025-04-30T06:43:19.758ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0379d704-bbfb-447f-8515-2f6141a3b1cc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94062025-04-30T06:43:19.758ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
94072025-04-30T06:43:19.762ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d342ceb-da29-4bb8-b242-fc107522dc78", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94082025-04-30T06:43:19.762ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
94092025-04-30T06:43:19.765ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
94102025-04-30T06:43:19.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95e41951-83d4-43ad-a5ad-37244fce356a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94112025-04-30T06:43:19.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
94122025-04-30T06:43:19.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceebcb8c-6539-4e77-8b59-ca4d354891e2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94132025-04-30T06:43:19.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
94142025-04-30T06:43:19.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a02eb8e-766c-4fc5-8ac8-5d500e2247b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94152025-04-30T06:43:19.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
94162025-04-30T06:43:19.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "097cd331-db10-4369-a045-3cd6cdd04612", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94172025-04-30T06:43:19.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
94182025-04-30T06:43:19.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1bc7dc8-5dca-4e70-9b80-ce8bd11cd0a1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94192025-04-30T06:43:19.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
94202025-04-30T06:43:19.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd7dd69-1279-4aee-8253-2e6069b053d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94212025-04-30T06:43:19.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
94222025-04-30T06:43:19.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "401fc765-d710-4947-943c-94013cf66044", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94232025-04-30T06:43:19.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
94242025-04-30T06:43:19.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "688f9b89-cccd-452b-89a0-c8c33bc75d1c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94252025-04-30T06:43:19.784ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
94262025-04-30T06:43:19.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8c67c03-311f-4990-81cf-7be24d712a00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94272025-04-30T06:43:19.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
94282025-04-30T06:43:19.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7382f6a9-2f47-4717-a646-880fd93e947b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94292025-04-30T06:43:19.788ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94302025-04-30T06:43:19.788ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94312025-04-30T06:43:19.788ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94322025-04-30T06:43:19.788ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94332025-04-30T06:43:19.789ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94342025-04-30T06:43:19.821ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94352025-04-30T06:43:19.821ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94362025-04-30T06:43:19.821ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94372025-04-30T06:43:19.821ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
94382025-04-30T06:43:19.823ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94392025-04-30T06:43:19.823ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94402025-04-30T06:43:19.826ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94412025-04-30T06:43:19.826ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94422025-04-30T06:43:19.826ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94432025-04-30T06:43:19.826ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94442025-04-30T06:43:19.827ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94452025-04-30T06:43:19.827ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94462025-04-30T06:43:19.837ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94472025-04-30T06:43:19.849ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94482025-04-30T06:43:19.849ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94492025-04-30T06:43:19.849ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
94502025-04-30T06:43:19.850ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94512025-04-30T06:43:19.851ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94522025-04-30T06:43:19.856ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94532025-04-30T06:43:19.856ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94542025-04-30T06:43:19.856ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94552025-04-30T06:43:19.857ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94562025-04-30T06:43:19.858ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94572025-04-30T06:43:19.858ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
94582025-04-30T06:43:19.859ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94592025-04-30T06:43:19.989ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
94602025-04-30T06:43:19.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
94612025-04-30T06:43:19.990ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:43117 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94622025-04-30T06:43:19.990ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94632025-04-30T06:43:19.990ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94642025-04-30T06:43:19.990ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
94652025-04-30T06:43:19.992ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
94662025-04-30T06:43:19.995ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:43117 req_id = 587b5a0c-0c7a-4d57-8aac-8f18b82949f5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94672025-04-30T06:43:19.996ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 501 local_addr = [::1]:45714 method = GET remote_addr = [::1]:43117 req_id = 587b5a0c-0c7a-4d57-8aac-8f18b82949f5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94682025-04-30T06:43:19.996ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "587b5a0c-0c7a-4d57-8aac-8f18b82949f5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
94692025-04-30T06:43:19.999ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94702025-04-30T06:43:19.999ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94712025-04-30T06:43:19.999ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
94722025-04-30T06:43:20.003ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94732025-04-30T06:43:20.003ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94742025-04-30T06:43:20.003ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94752025-04-30T06:43:20.003ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94762025-04-30T06:43:20.004ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94772025-04-30T06:43:20.010ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
94782025-04-30T06:43:20.010ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
94792025-04-30T06:43:20.012ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94802025-04-30T06:43:20.017ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94812025-04-30T06:43:20.017ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94822025-04-30T06:43:20.017ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94832025-04-30T06:43:20.018ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94842025-04-30T06:43:20.023ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
94852025-04-30T06:43:20.023ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94862025-04-30T06:43:20.059ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
94872025-04-30T06:43:20.068ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
94882025-04-30T06:43:20.068ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
94892025-04-30T06:43:20.068ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 593.191586ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
94902025-04-30T06:43:20.068ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
94912025-04-30T06:43:20.068ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94922025-04-30T06:43:20.068ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94932025-04-30T06:43:20.069ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94942025-04-30T06:43:20.069ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94952025-04-30T06:43:20.070ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94962025-04-30T06:43:20.074ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94972025-04-30T06:43:20.080ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94982025-04-30T06:43:20.080ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94992025-04-30T06:43:20.081ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95002025-04-30T06:43:20.081ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95012025-04-30T06:43:20.084ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
95022025-04-30T06:43:20.084ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95032025-04-30T06:43:20.085ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 96 b2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63258 req_id = be20b67f-2fa4-4019-b016-67dc60465f4b
95042025-04-30T06:43:20.085ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38578,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63258 req_id = be20b67f-2fa4-4019-b016-67dc60465f4b
95052025-04-30T06:43:20.086ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
95062025-04-30T06:43:20.086ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95072025-04-30T06:43:20.086ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63258 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = be20b67f-2fa4-4019-b016-67dc60465f4b
95082025-04-30T06:43:20.086ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4264, tv_nsec: 885522341 } })
95092025-04-30T06:43:20.086ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
95102025-04-30T06:43:20.087ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d1 aa 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58024 req_id = e7aeefa8-c719-4307-9224-a3387bf503b2
95112025-04-30T06:43:20.087ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53674,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58024 req_id = e7aeefa8-c719-4307-9224-a3387bf503b2
95122025-04-30T06:43:20.087ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95132025-04-30T06:43:20.087ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58024 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e7aeefa8-c719-4307-9224-a3387bf503b2
95142025-04-30T06:43:20.156ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95152025-04-30T06:43:20.156ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95162025-04-30T06:43:20.156ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95172025-04-30T06:43:20.156ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95182025-04-30T06:43:20.157ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95192025-04-30T06:43:20.158ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95202025-04-30T06:43:20.159ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
95212025-04-30T06:43:20.161ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 69 0e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c i............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60408 req_id = 37c866cf-c8fe-409d-a30f-b91f568e5d4d
95222025-04-30T06:43:20.161ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26894,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60408 req_id = 37c866cf-c8fe-409d-a30f-b91f568e5d4d
95232025-04-30T06:43:20.161ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
95242025-04-30T06:43:20.161ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26894, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60408 req_id = 37c866cf-c8fe-409d-a30f-b91f568e5d4d
95252025-04-30T06:43:20.161ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
95262025-04-30T06:43:20.161ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
95272025-04-30T06:43:20.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
95282025-04-30T06:43:20.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "75ea65d2-7e75-4765-b06c-f7b9367844c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
95292025-04-30T06:43:20.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
95302025-04-30T06:43:20.163ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95312025-04-30T06:43:20.169ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95322025-04-30T06:43:20.169ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95332025-04-30T06:43:20.169ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95342025-04-30T06:43:20.170ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95352025-04-30T06:43:20.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12425746-a9f8-408e-978c-ba7896cfe084", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:19 GMT"} })
95362025-04-30T06:43:20.173ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
95372025-04-30T06:43:20.173ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95382025-04-30T06:43:20.173ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95392025-04-30T06:43:20.174ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95402025-04-30T06:43:20.183ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f435255-5ac2-4f6c-a2cf-b972b9486f75", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95412025-04-30T06:43:20.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
95422025-04-30T06:43:20.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29616609-478b-4a09-b255-00903c08b7cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95432025-04-30T06:43:20.191ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
95442025-04-30T06:43:20.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "040040e3-f48a-4f74-a85a-c93dacb703de", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95452025-04-30T06:43:20.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
95462025-04-30T06:43:20.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb3baa2-c5f5-4a66-b973-7e5b04eb71da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95472025-04-30T06:43:20.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
95482025-04-30T06:43:20.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fa8ab8f-b53d-4c2e-915c-79dbdb5ac8ca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95492025-04-30T06:43:20.209ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
95502025-04-30T06:43:20.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbdcd0ed-08d4-46b1-b53a-bb833ec02925", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95512025-04-30T06:43:20.212ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
95522025-04-30T06:43:20.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d308ef0d-317d-4c2f-bb95-702134476431", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95532025-04-30T06:43:20.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
95542025-04-30T06:43:20.216ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
95552025-04-30T06:43:20.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7250e27e-6231-4637-9c05-e4c9a2181245", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95562025-04-30T06:43:20.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
95572025-04-30T06:43:20.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc54a5e-a30d-4734-aee2-b1641c553214", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95582025-04-30T06:43:20.220ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
95592025-04-30T06:43:20.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f88a6b3-4be2-438e-8614-de0ffb5bb892", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95602025-04-30T06:43:20.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
95612025-04-30T06:43:20.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e95bb4-52bf-47c4-88e8-4bb40184b610", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95622025-04-30T06:43:20.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
95632025-04-30T06:43:20.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58591f9a-9e64-4963-b485-f18db2313f44", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95642025-04-30T06:43:20.227ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
95652025-04-30T06:43:20.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c229c023-26a7-4cb7-a4f6-b7fcf924fdb1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95662025-04-30T06:43:20.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
95672025-04-30T06:43:20.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40343800-2177-4cbf-af0a-22947f96aed7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95682025-04-30T06:43:20.232ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
95692025-04-30T06:43:20.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88f804e5-c584-42a1-9d2e-6ffbccf80fcd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95702025-04-30T06:43:20.235ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
95712025-04-30T06:43:20.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd0c8274-1fe4-43ff-9aa5-bf769fe0677d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95722025-04-30T06:43:20.241ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
95732025-04-30T06:43:20.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce4eb26d-e57f-4dd5-be53-381511a91bed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95742025-04-30T06:43:20.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
95752025-04-30T06:43:20.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a985875-247a-4e14-82f8-908f42b584a5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95762025-04-30T06:43:20.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
95772025-04-30T06:43:20.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "204ff52d-c1c2-460b-a693-59478c313d37", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95782025-04-30T06:43:20.248ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
95792025-04-30T06:43:20.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3479043c-7ab2-4811-b07f-48dc8b674081", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95802025-04-30T06:43:20.252ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
95812025-04-30T06:43:20.255ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36c0f061-420c-465b-8e2c-73b02a1a1f02", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95822025-04-30T06:43:20.255ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
95832025-04-30T06:43:20.269ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95842025-04-30T06:43:20.269ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95852025-04-30T06:43:20.269ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95862025-04-30T06:43:20.269ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95872025-04-30T06:43:20.270ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95882025-04-30T06:43:20.271ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d2bdfc9-1bd0-463e-84bb-4b04a3b71038", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95892025-04-30T06:43:20.271ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
95902025-04-30T06:43:20.275ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95912025-04-30T06:43:20.281ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95922025-04-30T06:43:20.281ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95932025-04-30T06:43:20.281ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95942025-04-30T06:43:20.282ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95952025-04-30T06:43:20.292ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95962025-04-30T06:43:20.292ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
95972025-04-30T06:43:20.293ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95982025-04-30T06:43:20.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e5dafe-1f18-46a5-af51-751f31556292", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
95992025-04-30T06:43:20.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
96002025-04-30T06:43:20.312ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bee92a6-0397-4ee6-8bd9-a1b87b9195e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96012025-04-30T06:43:20.312ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
96022025-04-30T06:43:20.315ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f266e7-0312-461a-840b-9777a1c00da0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96032025-04-30T06:43:20.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
96042025-04-30T06:43:20.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ef03f63-b372-4632-8c54-a3848adc9654", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96052025-04-30T06:43:20.318ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
96062025-04-30T06:43:20.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25a7de77-395f-41e2-9515-6f74bc32e2b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96072025-04-30T06:43:20.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
96082025-04-30T06:43:20.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af295c58-6f9b-41f3-baa5-c3b8ad8a0706", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96092025-04-30T06:43:20.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
96102025-04-30T06:43:20.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "432b0d98-c186-4e0b-ae3e-8542934c8870", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96112025-04-30T06:43:20.326ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
96122025-04-30T06:43:20.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da3b628-b2cc-4814-8804-ce2982fc2c4b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96132025-04-30T06:43:20.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
96142025-04-30T06:43:20.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d979e8-9f50-4556-a2d2-7e7d0016f6dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96152025-04-30T06:43:20.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
96162025-04-30T06:43:20.334ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2869ca29-0a88-42c9-9802-4f030be7b241", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96172025-04-30T06:43:20.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
96182025-04-30T06:43:20.336ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
96192025-04-30T06:43:20.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa65c995-b207-4fa2-aa95-45f727661c1c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96202025-04-30T06:43:20.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
96212025-04-30T06:43:20.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9260be56-95b2-43ef-bc74-9f4e49bc33f5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96222025-04-30T06:43:20.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
96232025-04-30T06:43:20.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5eebe18-1ef8-4927-8f19-bc46b06eac22", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96242025-04-30T06:43:20.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
96252025-04-30T06:43:20.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8911a5a2-4262-4fdf-9ef0-4e83db4e18fb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96262025-04-30T06:43:20.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
96272025-04-30T06:43:20.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd802d1f-5680-4847-bf2e-b248a1c7e307", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96282025-04-30T06:43:20.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
96292025-04-30T06:43:20.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f11c1a-68eb-4b5f-ab69-d36f1aeeb679", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96302025-04-30T06:43:20.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
96312025-04-30T06:43:20.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8374ec35-3bb7-4b53-8d61-730297d9cda9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96322025-04-30T06:43:20.354ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
96332025-04-30T06:43:20.356ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c901ace-f737-43de-a662-8c4825f61427", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96342025-04-30T06:43:20.356ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
96352025-04-30T06:43:20.357ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8726916c-ffce-447d-8b89-676196e74c24", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96362025-04-30T06:43:20.357ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
96372025-04-30T06:43:20.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09eb1cc2-afb3-494d-9196-176525b15ce1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96382025-04-30T06:43:20.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
96392025-04-30T06:43:20.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70b5cda-ba6f-45ea-a3bd-62fe9cb3ee89", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96402025-04-30T06:43:20.361ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96412025-04-30T06:43:20.361ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96422025-04-30T06:43:20.361ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96432025-04-30T06:43:20.361ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96442025-04-30T06:43:20.362ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96452025-04-30T06:43:20.365ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96462025-04-30T06:43:20.370ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96472025-04-30T06:43:20.371ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96482025-04-30T06:43:20.371ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96492025-04-30T06:43:20.371ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96502025-04-30T06:43:20.371ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96512025-04-30T06:43:20.372ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96522025-04-30T06:43:20.375ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96532025-04-30T06:43:20.379ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96542025-04-30T06:43:20.379ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96552025-04-30T06:43:20.379ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96562025-04-30T06:43:20.380ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96572025-04-30T06:43:20.387ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96582025-04-30T06:43:20.387ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96592025-04-30T06:43:20.388ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96602025-04-30T06:43:20.388ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
96612025-04-30T06:43:20.388ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96622025-04-30T06:43:20.464ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
96632025-04-30T06:43:20.465ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
96642025-04-30T06:43:20.466ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:50502 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
96652025-04-30T06:43:20.468ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96662025-04-30T06:43:20.476ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96672025-04-30T06:43:20.476ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96682025-04-30T06:43:20.477ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
96692025-04-30T06:43:20.478ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96702025-04-30T06:43:20.478ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:50502 req_id = e0a5c506-3e6d-46ef-a289-0d39501f0f96 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96712025-04-30T06:43:20.482ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 4076 local_addr = [::1]:45714 method = GET remote_addr = [::1]:50502 req_id = e0a5c506-3e6d-46ef-a289-0d39501f0f96 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
96722025-04-30T06:43:20.486ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a5c506-3e6d-46ef-a289-0d39501f0f96", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
96732025-04-30T06:43:20.497ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96742025-04-30T06:43:20.497ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
96752025-04-30T06:43:20.497ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
96762025-04-30T06:43:20.501ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96772025-04-30T06:43:20.501ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96782025-04-30T06:43:20.501ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96792025-04-30T06:43:20.501ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96802025-04-30T06:43:20.502ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96812025-04-30T06:43:20.504ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96822025-04-30T06:43:20.505ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
96832025-04-30T06:43:20.505ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96842025-04-30T06:43:20.511ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96852025-04-30T06:43:20.524ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96862025-04-30T06:43:20.524ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96872025-04-30T06:43:20.524ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96882025-04-30T06:43:20.526ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96892025-04-30T06:43:20.578ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
96902025-04-30T06:43:20.582ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
96912025-04-30T06:43:20.582ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 513.831967ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
96922025-04-30T06:43:20.582ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
96932025-04-30T06:43:20.583ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
96942025-04-30T06:43:20.583ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96952025-04-30T06:43:20.583ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96962025-04-30T06:43:20.583ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96972025-04-30T06:43:20.584ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96982025-04-30T06:43:20.596ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
96992025-04-30T06:43:20.598ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97002025-04-30T06:43:20.609ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97012025-04-30T06:43:20.609ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97022025-04-30T06:43:20.609ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97032025-04-30T06:43:20.610ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97042025-04-30T06:43:20.615ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
97052025-04-30T06:43:20.615ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97062025-04-30T06:43:20.632ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6f de 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 o............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57785 req_id = f360929b-ac77-428c-80c8-741f71b56247
97072025-04-30T06:43:20.633ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28638,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57785 req_id = f360929b-ac77-428c-80c8-741f71b56247
97082025-04-30T06:43:20.633ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
97092025-04-30T06:43:20.633ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97102025-04-30T06:43:20.633ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57785 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f360929b-ac77-428c-80c8-741f71b56247
97112025-04-30T06:43:20.635ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4265, tv_nsec: 434563711 } })
97122025-04-30T06:43:20.635ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
97132025-04-30T06:43:20.636ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d d5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56216 req_id = 20a2d634-1349-4a67-8192-3a0a2b9833cb
97142025-04-30T06:43:20.636ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32213,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56216 req_id = 20a2d634-1349-4a67-8192-3a0a2b9833cb
97152025-04-30T06:43:20.636ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97162025-04-30T06:43:20.636ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56216 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 20a2d634-1349-4a67-8192-3a0a2b9833cb
97172025-04-30T06:43:20.721ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97182025-04-30T06:43:20.722ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97192025-04-30T06:43:20.722ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97202025-04-30T06:43:20.723ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97212025-04-30T06:43:20.723ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
97222025-04-30T06:43:20.724ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97232025-04-30T06:43:20.724ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97242025-04-30T06:43:20.724ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97252025-04-30T06:43:20.725ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 84 2f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ./..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64254 req_id = dacf1932-5a06-4907-9b50-504ce6c5a801
97262025-04-30T06:43:20.725ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33839,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64254 req_id = dacf1932-5a06-4907-9b50-504ce6c5a801
97272025-04-30T06:43:20.726ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
97282025-04-30T06:43:20.726ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33839, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64254 req_id = dacf1932-5a06-4907-9b50-504ce6c5a801
97292025-04-30T06:43:20.731ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
97302025-04-30T06:43:20.731ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
97312025-04-30T06:43:20.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
97322025-04-30T06:43:20.731ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97332025-04-30T06:43:20.733ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97342025-04-30T06:43:20.733ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97352025-04-30T06:43:20.733ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
97362025-04-30T06:43:20.733ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97372025-04-30T06:43:20.747ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97382025-04-30T06:43:20.747ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97392025-04-30T06:43:20.747ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
97402025-04-30T06:43:20.748ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97412025-04-30T06:43:20.752ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97422025-04-30T06:43:20.752ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
97432025-04-30T06:43:20.760ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97442025-04-30T06:43:20.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d8faf8a-2303-4dd3-b547-91e744808dd2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97452025-04-30T06:43:20.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
97462025-04-30T06:43:20.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b45f889b-18a9-40b2-8cf0-a2789fbe5f89", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97472025-04-30T06:43:20.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
97482025-04-30T06:43:20.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a841e926-b776-4eb2-8be3-84278f1cb245", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97492025-04-30T06:43:20.776ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
97502025-04-30T06:43:20.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f23c53a-b091-4f20-a93d-ed080f1f254c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97512025-04-30T06:43:20.779ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
97522025-04-30T06:43:20.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2bb72b-7234-48b4-abdf-7b2469b9b7e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97532025-04-30T06:43:20.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
97542025-04-30T06:43:20.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc7a8421-01a8-4616-b09c-615cdd3f922a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97552025-04-30T06:43:20.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
97562025-04-30T06:43:20.797ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee4c0fd9-2354-4844-a67a-1e6d9e644ecd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97572025-04-30T06:43:20.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
97582025-04-30T06:43:20.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9667118-6448-4b18-b17f-8ccc19d39ecd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97592025-04-30T06:43:20.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
97602025-04-30T06:43:20.806ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b596d7e1-f3c1-4a6e-86b7-06ee1c70f4de", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97612025-04-30T06:43:20.806ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
97622025-04-30T06:43:20.808ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
97632025-04-30T06:43:20.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "979b8b61-b5db-4da8-a720-b357419fc1cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97642025-04-30T06:43:20.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
97652025-04-30T06:43:20.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ffdee8-f943-4c2e-8995-4ab64551a01a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97662025-04-30T06:43:20.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
97672025-04-30T06:43:20.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e40a3a-6296-4722-87a7-a3be86781cf5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97682025-04-30T06:43:20.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
97692025-04-30T06:43:20.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ab80aa2-ac6f-4023-9d38-39f93953e052", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97702025-04-30T06:43:20.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
97712025-04-30T06:43:20.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a438cfa-97af-4ef8-9cd4-d79c40e24ef3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97722025-04-30T06:43:20.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
97732025-04-30T06:43:20.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9955ce-b774-4787-bdbe-508e726f98dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97742025-04-30T06:43:20.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
97752025-04-30T06:43:20.845ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ffc4de3-2994-4d0d-80d2-cf04a1aa1933", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97762025-04-30T06:43:20.845ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
97772025-04-30T06:43:20.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3314473d-f98a-4222-8634-50599284b597", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97782025-04-30T06:43:20.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
97792025-04-30T06:43:20.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac4a6863-52e7-4f80-8675-0da8875fcd12", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97802025-04-30T06:43:20.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
97812025-04-30T06:43:20.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a1aed3-8091-4258-96ee-6158e1241434", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97822025-04-30T06:43:20.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
97832025-04-30T06:43:20.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d4b4ff6-3a87-4b94-bc69-f5952224897f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97842025-04-30T06:43:20.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
97852025-04-30T06:43:20.860ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97862025-04-30T06:43:20.860ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97872025-04-30T06:43:20.860ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97882025-04-30T06:43:20.860ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
97892025-04-30T06:43:20.861ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97902025-04-30T06:43:20.866ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60bcc500-919d-44ca-a031-a93a4fbba4a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97912025-04-30T06:43:20.866ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
97922025-04-30T06:43:20.867ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97932025-04-30T06:43:20.879ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97942025-04-30T06:43:20.879ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97952025-04-30T06:43:20.879ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
97962025-04-30T06:43:20.879ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97972025-04-30T06:43:20.882ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fff1295-7d87-4930-ac5f-f2090ba531c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
97982025-04-30T06:43:20.883ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
97992025-04-30T06:43:20.884ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98002025-04-30T06:43:20.884ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
98012025-04-30T06:43:20.885ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98022025-04-30T06:43:20.886ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b44cf548-69cb-4646-a341-26872ce1f624", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98032025-04-30T06:43:20.886ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
98042025-04-30T06:43:20.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc90d74-1362-4426-b06c-1211d45c282a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98052025-04-30T06:43:20.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
98062025-04-30T06:43:20.892ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "965ee8f4-3f4d-4ec5-a61d-2cb50ca19fd2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98072025-04-30T06:43:20.892ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
98082025-04-30T06:43:20.902ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "601ca5ec-0c3e-489d-9d1e-7c922635526c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98092025-04-30T06:43:20.909ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
98102025-04-30T06:43:20.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76707db3-5f25-46ee-a337-6a9dc176c50f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98112025-04-30T06:43:20.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
98122025-04-30T06:43:20.921ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "358643cb-7565-4e3c-81a1-494bc31b6bb7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98132025-04-30T06:43:20.921ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
98142025-04-30T06:43:20.923ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aee2d417-248f-4e9d-ae54-c0fe2adfe7ee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98152025-04-30T06:43:20.923ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
98162025-04-30T06:43:20.931ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
98172025-04-30T06:43:20.937ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e33bff4-c052-40b0-a52b-06507767b7e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98182025-04-30T06:43:20.937ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
98192025-04-30T06:43:20.939ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7a589de-5b4d-495e-b732-1360061c7fe7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98202025-04-30T06:43:20.940ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
98212025-04-30T06:43:20.943ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59b42a5a-c902-4a00-887f-a338ec945e69", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98222025-04-30T06:43:20.943ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
98232025-04-30T06:43:20.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee1f49d-d1c6-4982-8a51-691b84acb96f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98242025-04-30T06:43:20.946ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
98252025-04-30T06:43:20.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f5e49f-f2a4-40fb-9471-6af87589bea9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98262025-04-30T06:43:20.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
98272025-04-30T06:43:20.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9764a502-c9a9-4bf7-a9c1-1c2b6c259ca5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98282025-04-30T06:43:20.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
98292025-04-30T06:43:20.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ead696-6c19-42b0-ae4e-d6ef0e6fa26a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98302025-04-30T06:43:20.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
98312025-04-30T06:43:20.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fc34f62-b28f-45cd-ad23-b8f0d19f9b21", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98322025-04-30T06:43:20.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
98332025-04-30T06:43:20.959ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f33b733-6ae1-416e-8b5d-b3db04e24947", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98342025-04-30T06:43:20.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
98352025-04-30T06:43:20.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53626cd4-66bd-40fc-9199-07a8870cc7b7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98362025-04-30T06:43:20.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
98372025-04-30T06:43:20.972ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a559f45-4b46-43be-8229-5343c8f1bb95", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98382025-04-30T06:43:20.972ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
98392025-04-30T06:43:20.974ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98402025-04-30T06:43:20.974ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98412025-04-30T06:43:20.975ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
98422025-04-30T06:43:20.976ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
98432025-04-30T06:43:20.977ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85fa0801-6fc0-42c6-a995-8034cf6cfb59", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98442025-04-30T06:43:20.977ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
98452025-04-30T06:43:20.978ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a77e252a-6786-4ae8-9852-94d8a6a50990", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98462025-04-30T06:43:20.978ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
98472025-04-30T06:43:20.980ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
98482025-04-30T06:43:20.980ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
98492025-04-30T06:43:20.982ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d71ba6-172a-4428-b8f9-700c013d6781", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98502025-04-30T06:43:20.982ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
98512025-04-30T06:43:20.983ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98522025-04-30T06:43:20.984ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98532025-04-30T06:43:20.984ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98542025-04-30T06:43:20.984ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
98552025-04-30T06:43:20.985ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98562025-04-30T06:43:20.991ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
98572025-04-30T06:43:20.991ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98582025-04-30T06:43:20.992ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98592025-04-30T06:43:21.002ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98602025-04-30T06:43:21.002ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98612025-04-30T06:43:21.002ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
98622025-04-30T06:43:21.003ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98632025-04-30T06:43:21.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea63f33d-90d4-4f76-8765-3408cd477944", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98642025-04-30T06:43:21.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
98652025-04-30T06:43:21.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ffd40cb-acda-4a95-a436-e0dc19232de2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98662025-04-30T06:43:21.005ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98672025-04-30T06:43:21.005ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98682025-04-30T06:43:21.005ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98692025-04-30T06:43:21.005ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98702025-04-30T06:43:21.006ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98712025-04-30T06:43:21.006ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98722025-04-30T06:43:21.006ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
98732025-04-30T06:43:21.007ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98742025-04-30T06:43:21.009ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98752025-04-30T06:43:21.014ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98762025-04-30T06:43:21.014ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98772025-04-30T06:43:21.014ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98782025-04-30T06:43:21.014ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98792025-04-30T06:43:21.015ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98802025-04-30T06:43:21.015ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98812025-04-30T06:43:21.018ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98822025-04-30T06:43:21.025ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
98832025-04-30T06:43:21.025ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98842025-04-30T06:43:21.025ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98852025-04-30T06:43:21.026ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98862025-04-30T06:43:21.098ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
98872025-04-30T06:43:21.098ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
98882025-04-30T06:43:21.099ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:33353 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
98892025-04-30T06:43:21.099ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
98902025-04-30T06:43:21.100ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:33353 req_id = 6f499b8d-4b9a-49ae-9f2e-c3f4e9c20976 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98912025-04-30T06:43:21.100ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 572 local_addr = [::1]:45714 method = GET remote_addr = [::1]:33353 req_id = 6f499b8d-4b9a-49ae-9f2e-c3f4e9c20976 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98922025-04-30T06:43:21.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f499b8d-4b9a-49ae-9f2e-c3f4e9c20976", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:20 GMT"} })
98932025-04-30T06:43:21.108ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98942025-04-30T06:43:21.108ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98952025-04-30T06:43:21.108ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
98962025-04-30T06:43:21.119ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98972025-04-30T06:43:21.120ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98982025-04-30T06:43:21.120ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98992025-04-30T06:43:21.120ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99002025-04-30T06:43:21.121ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99012025-04-30T06:43:21.128ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99022025-04-30T06:43:21.133ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99032025-04-30T06:43:21.133ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99042025-04-30T06:43:21.133ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99052025-04-30T06:43:21.134ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99062025-04-30T06:43:21.152ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
99072025-04-30T06:43:21.171ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99082025-04-30T06:43:21.171ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99092025-04-30T06:43:21.171ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 588.658152ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99102025-04-30T06:43:21.171ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99112025-04-30T06:43:21.171ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99122025-04-30T06:43:21.172ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99132025-04-30T06:43:21.172ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99142025-04-30T06:43:21.172ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99152025-04-30T06:43:21.175ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99162025-04-30T06:43:21.178ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99172025-04-30T06:43:21.183ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99182025-04-30T06:43:21.183ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99192025-04-30T06:43:21.183ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99202025-04-30T06:43:21.184ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99212025-04-30T06:43:21.188ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
99222025-04-30T06:43:21.188ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99232025-04-30T06:43:21.189ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9c fa 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49748 req_id = 2376d529-7386-4427-a466-7d6969973e20
99242025-04-30T06:43:21.189ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40186,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49748 req_id = 2376d529-7386-4427-a466-7d6969973e20
99252025-04-30T06:43:21.190ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
99262025-04-30T06:43:21.191ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99272025-04-30T06:43:21.191ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49748 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2376d529-7386-4427-a466-7d6969973e20
99282025-04-30T06:43:21.191ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4265, tv_nsec: 990684505 } })
99292025-04-30T06:43:21.191ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
99302025-04-30T06:43:21.192ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fa c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57360 req_id = 7ed9e7e2-7568-4182-bb34-31a5c7978251
99312025-04-30T06:43:21.192ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64195,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57360 req_id = 7ed9e7e2-7568-4182-bb34-31a5c7978251
99322025-04-30T06:43:21.192ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99332025-04-30T06:43:21.192ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57360 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7ed9e7e2-7568-4182-bb34-31a5c7978251
99342025-04-30T06:43:21.253ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99352025-04-30T06:43:21.253ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99362025-04-30T06:43:21.253ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99372025-04-30T06:43:21.253ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99382025-04-30T06:43:21.253ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
99392025-04-30T06:43:21.254ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99402025-04-30T06:43:21.255ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 64 62 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c db..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65074 req_id = 17b44383-7dc4-42bb-acef-7f40532f3bca
99412025-04-30T06:43:21.255ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25698,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65074 req_id = 17b44383-7dc4-42bb-acef-7f40532f3bca
99422025-04-30T06:43:21.256ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
99432025-04-30T06:43:21.256ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25698, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65074 req_id = 17b44383-7dc4-42bb-acef-7f40532f3bca
99442025-04-30T06:43:21.257ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
99452025-04-30T06:43:21.257ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
99462025-04-30T06:43:21.258ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
99472025-04-30T06:43:21.259ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99482025-04-30T06:43:21.269ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99492025-04-30T06:43:21.269ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99502025-04-30T06:43:21.269ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
99512025-04-30T06:43:21.270ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99522025-04-30T06:43:21.285ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99532025-04-30T06:43:21.285ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
99542025-04-30T06:43:21.286ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99552025-04-30T06:43:21.298ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "79733e03-675e-420c-bd57-edb950ece8fc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99562025-04-30T06:43:21.298ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
99572025-04-30T06:43:21.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85427fb9-2f70-4022-9fea-6d148957865d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99582025-04-30T06:43:21.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
99592025-04-30T06:43:21.332ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e23d66-1ca8-4a87-875f-a490a8c67b25", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99602025-04-30T06:43:21.332ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
99612025-04-30T06:43:21.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c32d5390-dce4-4ec2-b49f-c7008e322b6a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99622025-04-30T06:43:21.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
99632025-04-30T06:43:21.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "073c96e6-24d6-4c17-b636-71d3e1536660", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99642025-04-30T06:43:21.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
99652025-04-30T06:43:21.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2449e0ca-8561-4193-8b46-0d3a7175698d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99662025-04-30T06:43:21.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
99672025-04-30T06:43:21.349ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f195aa23-f995-4037-94e2-56c38fa3dca1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99682025-04-30T06:43:21.349ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
99692025-04-30T06:43:21.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "564ca05f-7a91-45ff-996a-4ed7400e2841", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99702025-04-30T06:43:21.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
99712025-04-30T06:43:21.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e21726e0-fd49-4bfa-b8d1-fd960898cba3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99722025-04-30T06:43:21.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
99732025-04-30T06:43:21.356ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
99742025-04-30T06:43:21.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e87fcc4b-33c7-436d-9dd6-dc8ca530d616", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99752025-04-30T06:43:21.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
99762025-04-30T06:43:21.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4543d894-2cef-4fde-9f24-2d78f14c96f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99772025-04-30T06:43:21.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
99782025-04-30T06:43:21.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8414022-82a7-4f6d-9687-82bb0fbe2d86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99792025-04-30T06:43:21.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
99802025-04-30T06:43:21.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2a86641-cbcf-4f03-98c4-d6723c3bc49e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99812025-04-30T06:43:21.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
99822025-04-30T06:43:21.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a7077de-393e-4443-9ceb-d9e3c277a924", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99832025-04-30T06:43:21.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
99842025-04-30T06:43:21.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b47c709-4bcf-4b6d-9888-6b619b055126", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99852025-04-30T06:43:21.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
99862025-04-30T06:43:21.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71d64752-73e9-479e-9603-3782ccc7698b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99872025-04-30T06:43:21.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
99882025-04-30T06:43:21.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "680d381a-76a4-4290-926a-b6e9b4bfb0d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99892025-04-30T06:43:21.371ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
99902025-04-30T06:43:21.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad3bdbda-5704-4d9a-a4d4-7c81ed618fe3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99912025-04-30T06:43:21.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
99922025-04-30T06:43:21.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa780c6c-bf3e-4777-8093-01da368f1cf2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99932025-04-30T06:43:21.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
99942025-04-30T06:43:21.377ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf5424d3-7822-46ad-a046-18179febb0c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99952025-04-30T06:43:21.377ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
99962025-04-30T06:43:21.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e8034ae-e364-452b-af5c-c2d9329be67e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99972025-04-30T06:43:21.379ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
99982025-04-30T06:43:21.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87136848-c055-4cf6-87d7-6a2afe3e1262", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
99992025-04-30T06:43:21.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
100002025-04-30T06:43:21.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76a33d96-a4bf-4784-b4d0-d39d7394f8bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100012025-04-30T06:43:21.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
100022025-04-30T06:43:21.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d9ebff-3f1b-4246-ad2e-cd9c5c167212", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100032025-04-30T06:43:21.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
100042025-04-30T06:43:21.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "127b7f34-5296-456e-b317-c32b2c708b4c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100052025-04-30T06:43:21.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
100062025-04-30T06:43:21.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5701e871-dad2-4c6a-a710-7f792220f3aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100072025-04-30T06:43:21.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
100082025-04-30T06:43:21.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2550e82a-ae2d-4c99-93d3-47f12ede116f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100092025-04-30T06:43:21.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
100102025-04-30T06:43:21.393ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a30cc2b5-58cd-4691-8a05-bcafa773ab85", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100112025-04-30T06:43:21.394ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
100122025-04-30T06:43:21.395ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b8d2921-5f55-4dbc-8cc4-7e6b1534ef35", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100132025-04-30T06:43:21.396ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
100142025-04-30T06:43:21.398ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2be02d-6115-4402-91b9-6c5ee338f3db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100152025-04-30T06:43:21.398ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
100162025-04-30T06:43:21.399ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "708d6b9e-71ac-4d93-8e90-a4f10c38ecb4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100172025-04-30T06:43:21.399ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
100182025-04-30T06:43:21.401ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd3abe82-bb1a-41d6-8df8-0867e2393fab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100192025-04-30T06:43:21.401ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
100202025-04-30T06:43:21.402ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dc2240e-b758-4200-9fc3-63aeb404ba7c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100212025-04-30T06:43:21.403ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
100222025-04-30T06:43:21.404ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bec7253-b34a-417c-98c5-35643c4c945c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100232025-04-30T06:43:21.404ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
100242025-04-30T06:43:21.405ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69789fbf-491d-40eb-897a-7f55ae7b14cf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100252025-04-30T06:43:21.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
100262025-04-30T06:43:21.406ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100272025-04-30T06:43:21.406ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100282025-04-30T06:43:21.406ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100292025-04-30T06:43:21.406ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
100302025-04-30T06:43:21.407ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100312025-04-30T06:43:21.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa9d814-d55b-43d7-9b18-d786f277f626", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100322025-04-30T06:43:21.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
100332025-04-30T06:43:21.410ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100342025-04-30T06:43:21.418ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100352025-04-30T06:43:21.418ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100362025-04-30T06:43:21.418ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
100372025-04-30T06:43:21.418ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100382025-04-30T06:43:21.422ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c858601-28ca-4c97-83cd-3dba8fb9844d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100392025-04-30T06:43:21.423ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
100402025-04-30T06:43:21.424ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100412025-04-30T06:43:21.424ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
100422025-04-30T06:43:21.426ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100432025-04-30T06:43:21.432ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2893c646-b539-4097-a811-2f35a03d1922", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100442025-04-30T06:43:21.432ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
100452025-04-30T06:43:21.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1fffeb3-7033-45c1-869b-1f92724ca3f6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100462025-04-30T06:43:21.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
100472025-04-30T06:43:21.441ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "158844d6-42e6-4470-9ce6-b3b94bf62704", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100482025-04-30T06:43:21.441ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
100492025-04-30T06:43:21.445ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f27346-c376-4b6f-b482-7c3b7685254d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100502025-04-30T06:43:21.446ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
100512025-04-30T06:43:21.447ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0879fb0-9d7b-4125-8424-768cfe0bef01", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100522025-04-30T06:43:21.448ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
100532025-04-30T06:43:21.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb70f4f-cb25-4dc2-8f1f-26d03b23e7fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100542025-04-30T06:43:21.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
100552025-04-30T06:43:21.452ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69597efb-1415-46df-9b09-ca30b28f1693", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100562025-04-30T06:43:21.452ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
100572025-04-30T06:43:21.463ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdbef2fc-590f-4c18-b261-229aa4a123de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100582025-04-30T06:43:21.463ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100592025-04-30T06:43:21.463ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100602025-04-30T06:43:21.463ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100612025-04-30T06:43:21.463ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100622025-04-30T06:43:21.465ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100632025-04-30T06:43:21.468ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100642025-04-30T06:43:21.479ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100652025-04-30T06:43:21.479ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100662025-04-30T06:43:21.480ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100672025-04-30T06:43:21.480ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100682025-04-30T06:43:21.480ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100692025-04-30T06:43:21.481ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100702025-04-30T06:43:21.488ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
100712025-04-30T06:43:21.489ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100722025-04-30T06:43:21.493ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100732025-04-30T06:43:21.493ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100742025-04-30T06:43:21.493ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100752025-04-30T06:43:21.494ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100762025-04-30T06:43:21.589ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
100772025-04-30T06:43:21.589ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
100782025-04-30T06:43:21.590ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100792025-04-30T06:43:21.590ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100802025-04-30T06:43:21.590ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100812025-04-30T06:43:21.590ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
100822025-04-30T06:43:21.591ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100832025-04-30T06:43:21.591ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:36068 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
100842025-04-30T06:43:21.593ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:36068 req_id = be0d5ed5-c3ce-4530-b1e6-e7892d449426 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100852025-04-30T06:43:21.594ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1159 local_addr = [::1]:45714 method = GET remote_addr = [::1]:36068 req_id = be0d5ed5-c3ce-4530-b1e6-e7892d449426 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100862025-04-30T06:43:21.595ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "be0d5ed5-c3ce-4530-b1e6-e7892d449426", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
100872025-04-30T06:43:21.599ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100882025-04-30T06:43:21.599ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100892025-04-30T06:43:21.599ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
100902025-04-30T06:43:21.602ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100912025-04-30T06:43:21.602ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100922025-04-30T06:43:21.603ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100932025-04-30T06:43:21.603ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100942025-04-30T06:43:21.605ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100952025-04-30T06:43:21.606ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100962025-04-30T06:43:21.613ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100972025-04-30T06:43:21.613ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100982025-04-30T06:43:21.613ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
100992025-04-30T06:43:21.613ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101002025-04-30T06:43:21.618ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101012025-04-30T06:43:21.622ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101022025-04-30T06:43:21.622ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101032025-04-30T06:43:21.622ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101042025-04-30T06:43:21.626ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101052025-04-30T06:43:21.630ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101062025-04-30T06:43:21.631ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
101072025-04-30T06:43:21.635ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101082025-04-30T06:43:21.639ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
101092025-04-30T06:43:21.639ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101102025-04-30T06:43:21.640ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = e90641e4-f745-419c-b992-4129bda956fa uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101112025-04-30T06:43:21.640ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = e90641e4-f745-419c-b992-4129bda956fa resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101122025-04-30T06:43:21.640ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = e90641e4-f745-419c-b992-4129bda956fa roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101132025-04-30T06:43:21.641ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = e90641e4-f745-419c-b992-4129bda956fa resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101142025-04-30T06:43:21.646ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 6523 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = e90641e4-f745-419c-b992-4129bda956fa response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101152025-04-30T06:43:21.647ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90641e4-f745-419c-b992-4129bda956fa", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101162025-04-30T06:43:21.647ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101172025-04-30T06:43:21.649ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2d95e0f7-30c5-433f-a49c-8df4378062f6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101182025-04-30T06:43:21.649ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2d95e0f7-30c5-433f-a49c-8df4378062f6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101192025-04-30T06:43:21.649ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2d95e0f7-30c5-433f-a49c-8df4378062f6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101202025-04-30T06:43:21.650ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2d95e0f7-30c5-433f-a49c-8df4378062f6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101212025-04-30T06:43:21.655ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 6828 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 2d95e0f7-30c5-433f-a49c-8df4378062f6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
101222025-04-30T06:43:21.656ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d95e0f7-30c5-433f-a49c-8df4378062f6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101232025-04-30T06:43:21.656ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101242025-04-30T06:43:21.656ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
101252025-04-30T06:43:21.656ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
101262025-04-30T06:43:21.668ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101272025-04-30T06:43:21.669ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101282025-04-30T06:43:21.669ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 497.855375ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101292025-04-30T06:43:21.669ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101302025-04-30T06:43:21.669ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101312025-04-30T06:43:21.670ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101322025-04-30T06:43:21.670ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101332025-04-30T06:43:21.670ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101342025-04-30T06:43:21.671ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101352025-04-30T06:43:21.676ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101362025-04-30T06:43:21.682ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101372025-04-30T06:43:21.682ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101382025-04-30T06:43:21.682ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101392025-04-30T06:43:21.683ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101402025-04-30T06:43:21.706ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101412025-04-30T06:43:21.707ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101422025-04-30T06:43:21.709ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
101432025-04-30T06:43:21.709ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101442025-04-30T06:43:21.711ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101452025-04-30T06:43:21.711ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101462025-04-30T06:43:21.711ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101472025-04-30T06:43:21.711ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
101482025-04-30T06:43:21.711ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101492025-04-30T06:43:21.713ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 84 05 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62251 req_id = 95b947b3-61a7-4687-a027-65a78ad56576
101502025-04-30T06:43:21.713ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33797,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62251 req_id = 95b947b3-61a7-4687-a027-65a78ad56576
101512025-04-30T06:43:21.714ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
101522025-04-30T06:43:21.714ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101532025-04-30T06:43:21.717ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62251 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 95b947b3-61a7-4687-a027-65a78ad56576
101542025-04-30T06:43:21.718ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4266, tv_nsec: 516926859 } })
101552025-04-30T06:43:21.718ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
101562025-04-30T06:43:21.718ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 98 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .t..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50633 req_id = 568cd166-acfe-497f-9608-23cee67f92ce
101572025-04-30T06:43:21.719ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39028,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50633 req_id = 568cd166-acfe-497f-9608-23cee67f92ce
101582025-04-30T06:43:21.719ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101592025-04-30T06:43:21.719ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50633 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 568cd166-acfe-497f-9608-23cee67f92ce
101602025-04-30T06:43:21.833ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101612025-04-30T06:43:21.833ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101622025-04-30T06:43:21.833ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101632025-04-30T06:43:21.833ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101642025-04-30T06:43:21.833ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
101652025-04-30T06:43:21.834ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101662025-04-30T06:43:21.835ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c7 46 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .F..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65087 req_id = e4791441-4bc3-46f5-8009-191427486c68
101672025-04-30T06:43:21.836ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51014,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65087 req_id = e4791441-4bc3-46f5-8009-191427486c68
101682025-04-30T06:43:21.836ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
101692025-04-30T06:43:21.838ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51014, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65087 req_id = e4791441-4bc3-46f5-8009-191427486c68
101702025-04-30T06:43:21.838ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
101712025-04-30T06:43:21.838ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
101722025-04-30T06:43:21.838ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
101732025-04-30T06:43:21.839ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101742025-04-30T06:43:21.847ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101752025-04-30T06:43:21.847ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101762025-04-30T06:43:21.847ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
101772025-04-30T06:43:21.849ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101782025-04-30T06:43:21.852ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101792025-04-30T06:43:21.852ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
101802025-04-30T06:43:21.853ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101812025-04-30T06:43:21.853ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "09398b9f-aabd-4856-9556-207e9df6aabe", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101822025-04-30T06:43:21.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
101832025-04-30T06:43:21.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51e2a925-c8f4-438c-810a-e57e3c0716fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101842025-04-30T06:43:21.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
101852025-04-30T06:43:21.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4c1d70c-fd15-4d64-b877-35527273b4b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101862025-04-30T06:43:21.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
101872025-04-30T06:43:21.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2257ee-b5fe-4233-b9fa-7a73ae7611f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101882025-04-30T06:43:21.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
101892025-04-30T06:43:21.923ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
101902025-04-30T06:43:21.927ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "927cebec-2ee8-4165-aae7-96ca6cb9d888", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101912025-04-30T06:43:21.928ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
101922025-04-30T06:43:21.938ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b70567-3277-4ad7-bbfa-1ce0fd991b50", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101932025-04-30T06:43:21.938ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
101942025-04-30T06:43:21.942ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b9aa596-3a76-4d93-9afd-7eafc814b903", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101952025-04-30T06:43:21.943ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
101962025-04-30T06:43:21.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7636acf-d88d-45a0-8a74-3e38750ff568", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101972025-04-30T06:43:21.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
101982025-04-30T06:43:21.952ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "adccce08-b198-405f-a0b2-5d6d536171ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
101992025-04-30T06:43:21.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
102002025-04-30T06:43:21.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9e3ffdd-53e9-4930-bdbd-b6be0383e5d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102012025-04-30T06:43:21.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
102022025-04-30T06:43:21.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f167167b-8ef1-4fa6-9c35-a5bbc4ff321a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102032025-04-30T06:43:21.956ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
102042025-04-30T06:43:21.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad1aa934-3519-4bfa-bc21-b682838bd28b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102052025-04-30T06:43:21.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
102062025-04-30T06:43:21.958ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "274b62a9-8237-4346-8f2d-beed2465f831", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102072025-04-30T06:43:21.959ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
102082025-04-30T06:43:21.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "144ece1d-4d69-47d1-b6a3-65131c14cdff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102092025-04-30T06:43:21.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
102102025-04-30T06:43:21.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddaac2fd-cac5-464f-8555-7e339cd83d3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102112025-04-30T06:43:21.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
102122025-04-30T06:43:21.979ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102132025-04-30T06:43:21.980ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102142025-04-30T06:43:21.980ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102152025-04-30T06:43:21.980ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
102162025-04-30T06:43:21.981ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102172025-04-30T06:43:21.981ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102182025-04-30T06:43:21.981ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102192025-04-30T06:43:21.993ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
102202025-04-30T06:43:21.996ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
102212025-04-30T06:43:22.003ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102222025-04-30T06:43:22.006ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102232025-04-30T06:43:22.006ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102242025-04-30T06:43:22.006ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
102252025-04-30T06:43:22.007ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102262025-04-30T06:43:22.016ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
102272025-04-30T06:43:22.016ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
102282025-04-30T06:43:22.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc28e679-bb00-46f0-86e9-25cb70a89226", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102292025-04-30T06:43:22.034ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
102302025-04-30T06:43:22.034ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102312025-04-30T06:43:22.034ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
102322025-04-30T06:43:22.035ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102332025-04-30T06:43:22.035ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
102342025-04-30T06:43:22.038ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102352025-04-30T06:43:22.042ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f28d85c-d419-4599-830f-eb2526ab94b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102362025-04-30T06:43:22.043ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
102372025-04-30T06:43:22.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7972b16a-f1ea-4dae-ae60-3c85b1395ad2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102382025-04-30T06:43:22.047ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
102392025-04-30T06:43:22.052ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5f8a47-5907-4beb-ac4a-b58968cfaef7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102402025-04-30T06:43:22.052ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
102412025-04-30T06:43:22.054ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43959c45-9bc4-42c0-88a7-eeb36803e4e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102422025-04-30T06:43:22.054ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
102432025-04-30T06:43:22.056ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "595e6380-19a6-4453-8048-d3ff5ec83714", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102442025-04-30T06:43:22.056ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
102452025-04-30T06:43:22.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20bab3b3-8d36-4d34-b227-b49cf22d34fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102462025-04-30T06:43:22.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
102472025-04-30T06:43:22.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c13ea0-bde3-4d65-87e5-812d9faa07aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102482025-04-30T06:43:22.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
102492025-04-30T06:43:22.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abc5f163-39bb-4799-84e2-c23b21aa0dc1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102502025-04-30T06:43:22.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
102512025-04-30T06:43:22.103ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
102522025-04-30T06:43:22.104ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
102532025-04-30T06:43:22.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d5a71d-b424-490d-8d40-5456b89b385c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102542025-04-30T06:43:22.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
102552025-04-30T06:43:22.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e67b6c3-1b15-4380-9397-043fedda2f56", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102562025-04-30T06:43:22.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
102572025-04-30T06:43:22.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "057a6427-9775-4750-83c7-8b30f88146d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102582025-04-30T06:43:22.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
102592025-04-30T06:43:22.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3addf2d1-be6a-46b0-b164-61eda226d9c4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102602025-04-30T06:43:22.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
102612025-04-30T06:43:22.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d7b2527-36d6-4953-9957-a52333581b00", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102622025-04-30T06:43:22.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
102632025-04-30T06:43:22.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1dadc2f-2a05-41ec-a1b3-8043a249da75", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102642025-04-30T06:43:22.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
102652025-04-30T06:43:22.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed60dd6b-79bf-4fe5-bc52-67338984d695", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102662025-04-30T06:43:22.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
102672025-04-30T06:43:22.133ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac94ad45-c2e7-4b51-93d6-918a9710a601", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102682025-04-30T06:43:22.133ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
102692025-04-30T06:43:22.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a99c007-d4f8-4d94-85c1-da13aeb154a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102702025-04-30T06:43:22.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
102712025-04-30T06:43:22.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "158a19fd-fe68-4a4f-93bc-a04107253a79", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102722025-04-30T06:43:22.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
102732025-04-30T06:43:22.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34f22942-a60d-4677-bf2b-8704b1894b5e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102742025-04-30T06:43:22.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
102752025-04-30T06:43:22.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f542d655-4af7-4044-bf29-6b41d3cdaf6c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102762025-04-30T06:43:22.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
102772025-04-30T06:43:22.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca16e2b-3728-4292-9e3b-cfbe0dd5f498", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102782025-04-30T06:43:22.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
102792025-04-30T06:43:22.144ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3d88775-cdc6-4e70-aaee-6fb7163ab577", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102802025-04-30T06:43:22.144ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
102812025-04-30T06:43:22.146ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01fc983f-7d3c-4ecc-a01e-dda76e357505", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102822025-04-30T06:43:22.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
102832025-04-30T06:43:22.149ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04deb167-9c23-4510-811f-bdc4fec99daf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102842025-04-30T06:43:22.149ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
102852025-04-30T06:43:22.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ffb26b-f9fd-46e6-9754-48d28ab8cd44", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102862025-04-30T06:43:22.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
102872025-04-30T06:43:22.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e45dac-7f70-43d5-ad34-24935eea1caa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102882025-04-30T06:43:22.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
102892025-04-30T06:43:22.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfdae1e6-6eeb-4e98-b9a3-74d1135fe026", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102902025-04-30T06:43:22.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
102912025-04-30T06:43:22.155ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102922025-04-30T06:43:22.155ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102932025-04-30T06:43:22.155ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102942025-04-30T06:43:22.155ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
102952025-04-30T06:43:22.156ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102962025-04-30T06:43:22.158ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59469faa-63ce-4154-860e-cc0607efdd71", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102972025-04-30T06:43:22.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
102982025-04-30T06:43:22.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b49a0bd9-0fe4-4c3f-b49e-f99d81668024", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:21 GMT"} })
102992025-04-30T06:43:22.166ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103002025-04-30T06:43:22.166ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103012025-04-30T06:43:22.166ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103022025-04-30T06:43:22.166ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103032025-04-30T06:43:22.167ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103042025-04-30T06:43:22.167ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103052025-04-30T06:43:22.175ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103062025-04-30T06:43:22.175ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103072025-04-30T06:43:22.175ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
103082025-04-30T06:43:22.176ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103092025-04-30T06:43:22.178ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103102025-04-30T06:43:22.181ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103112025-04-30T06:43:22.181ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103122025-04-30T06:43:22.181ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103132025-04-30T06:43:22.181ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103142025-04-30T06:43:22.181ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103152025-04-30T06:43:22.181ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103162025-04-30T06:43:22.185ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103172025-04-30T06:43:22.185ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
103182025-04-30T06:43:22.186ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103192025-04-30T06:43:22.188ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103202025-04-30T06:43:22.193ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103212025-04-30T06:43:22.202ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103222025-04-30T06:43:22.202ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103232025-04-30T06:43:22.203ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103242025-04-30T06:43:22.271ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
103252025-04-30T06:43:22.271ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
103262025-04-30T06:43:22.272ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:47316 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103272025-04-30T06:43:22.277ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:47316 req_id = 4cbe15df-c311-40a2-98ad-38ff087998f8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103282025-04-30T06:43:22.278ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 726 local_addr = [::1]:45714 method = GET remote_addr = [::1]:47316 req_id = 4cbe15df-c311-40a2-98ad-38ff087998f8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103292025-04-30T06:43:22.279ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cbe15df-c311-40a2-98ad-38ff087998f8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
103302025-04-30T06:43:22.282ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103312025-04-30T06:43:22.282ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103322025-04-30T06:43:22.282ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
103332025-04-30T06:43:22.284ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103342025-04-30T06:43:22.284ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103352025-04-30T06:43:22.284ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103362025-04-30T06:43:22.284ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103372025-04-30T06:43:22.284ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103382025-04-30T06:43:22.288ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103392025-04-30T06:43:22.291ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103402025-04-30T06:43:22.291ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103412025-04-30T06:43:22.291ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103422025-04-30T06:43:22.292ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103432025-04-30T06:43:22.372ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
103442025-04-30T06:43:22.418ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
103452025-04-30T06:43:22.418ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
103462025-04-30T06:43:22.418ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 748.773582ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
103472025-04-30T06:43:22.418ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
103482025-04-30T06:43:22.418ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103492025-04-30T06:43:22.418ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103502025-04-30T06:43:22.418ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103512025-04-30T06:43:22.418ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103522025-04-30T06:43:22.419ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103532025-04-30T06:43:22.425ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103542025-04-30T06:43:22.442ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103552025-04-30T06:43:22.442ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103562025-04-30T06:43:22.442ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103572025-04-30T06:43:22.442ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103582025-04-30T06:43:22.442ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103592025-04-30T06:43:22.443ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103602025-04-30T06:43:22.443ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103612025-04-30T06:43:22.443ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
103622025-04-30T06:43:22.443ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103632025-04-30T06:43:22.446ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103642025-04-30T06:43:22.467ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103652025-04-30T06:43:22.468ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103662025-04-30T06:43:22.468ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
103672025-04-30T06:43:22.468ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103682025-04-30T06:43:22.469ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
103692025-04-30T06:43:22.469ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
103702025-04-30T06:43:22.470ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: eb ba 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64993 req_id = 8cd00a4b-446e-4d19-af51-43ad24fc729a
103712025-04-30T06:43:22.471ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60346,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64993 req_id = 8cd00a4b-446e-4d19-af51-43ad24fc729a
103722025-04-30T06:43:22.471ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
103732025-04-30T06:43:22.471ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103742025-04-30T06:43:22.472ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64993 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8cd00a4b-446e-4d19-af51-43ad24fc729a
103752025-04-30T06:43:22.473ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4267, tv_nsec: 272654066 } })
103762025-04-30T06:43:22.473ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
103772025-04-30T06:43:22.474ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 17 fb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64143 req_id = 10d41310-0ed8-43e8-85c3-8270466117e4
103782025-04-30T06:43:22.474ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6139,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64143 req_id = 10d41310-0ed8-43e8-85c3-8270466117e4
103792025-04-30T06:43:22.475ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103802025-04-30T06:43:22.475ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64143 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 10d41310-0ed8-43e8-85c3-8270466117e4
103812025-04-30T06:43:22.476ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103822025-04-30T06:43:22.476ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
103832025-04-30T06:43:22.477ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103842025-04-30T06:43:22.651ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
103852025-04-30T06:43:22.653ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: be 18 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55577 req_id = 24783aa3-d4c6-441e-bd6e-886610eb3caa
103862025-04-30T06:43:22.654ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48664,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55577 req_id = 24783aa3-d4c6-441e-bd6e-886610eb3caa
103872025-04-30T06:43:22.654ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
103882025-04-30T06:43:22.654ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48664, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55577 req_id = 24783aa3-d4c6-441e-bd6e-886610eb3caa
103892025-04-30T06:43:22.655ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
103902025-04-30T06:43:22.655ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
103912025-04-30T06:43:22.655ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
103922025-04-30T06:43:22.660ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "74c4538b-8c17-4cf7-8dad-5fef86e5a8ea", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
103932025-04-30T06:43:22.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
103942025-04-30T06:43:22.664ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "555fdad8-14a8-40b7-b07d-56d8362937c5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
103952025-04-30T06:43:22.664ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
103962025-04-30T06:43:22.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdb6f4c0-89f3-4787-9a37-a6c03c2879d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
103972025-04-30T06:43:22.670ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
103982025-04-30T06:43:22.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d3efe28-d6de-4a62-8156-605fa9ea3603", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
103992025-04-30T06:43:22.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
104002025-04-30T06:43:22.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7f5991f-18e7-482f-823e-cbec07b1ba2f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104012025-04-30T06:43:22.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
104022025-04-30T06:43:22.699ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
104032025-04-30T06:43:22.705ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104042025-04-30T06:43:22.705ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104052025-04-30T06:43:22.708ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104062025-04-30T06:43:22.710ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104072025-04-30T06:43:22.710ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104082025-04-30T06:43:22.710ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104092025-04-30T06:43:22.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ce511f6-d404-49a7-b15a-38d130eb5ff1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104102025-04-30T06:43:22.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
104112025-04-30T06:43:22.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e7b2ae-8aa1-425a-99c3-b19475c1b65d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104122025-04-30T06:43:22.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
104132025-04-30T06:43:22.750ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104142025-04-30T06:43:22.750ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104152025-04-30T06:43:22.750ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104162025-04-30T06:43:22.750ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104172025-04-30T06:43:22.751ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104182025-04-30T06:43:22.754ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104192025-04-30T06:43:22.759ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104202025-04-30T06:43:22.759ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104212025-04-30T06:43:22.759ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104222025-04-30T06:43:22.759ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104232025-04-30T06:43:22.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f919c3-cc6d-45e3-9a67-c77cbc25ed47", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104242025-04-30T06:43:22.761ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
104252025-04-30T06:43:22.762ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104262025-04-30T06:43:22.762ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104272025-04-30T06:43:22.762ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104282025-04-30T06:43:22.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15dec81b-51a5-473d-9ccd-294535cacdf7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104292025-04-30T06:43:22.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
104302025-04-30T06:43:22.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f72e9a-49d0-4ce7-8e82-81e7e993c98a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104312025-04-30T06:43:22.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
104322025-04-30T06:43:22.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9403261d-19f6-4223-ad01-366f93776689", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104332025-04-30T06:43:22.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
104342025-04-30T06:43:22.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c85ba462-8842-4b23-b711-a808bb568702", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104352025-04-30T06:43:22.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
104362025-04-30T06:43:22.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a8375b-5811-4efc-bc2e-206d2d628884", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104372025-04-30T06:43:22.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
104382025-04-30T06:43:22.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49cf40f-1522-43ec-8b57-a295e2b9a9e4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104392025-04-30T06:43:22.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
104402025-04-30T06:43:22.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f790a19f-2495-4824-a83f-80b38650cd0d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104412025-04-30T06:43:22.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
104422025-04-30T06:43:22.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4e70d7-8096-49be-86e7-eeccd56fc1e6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104432025-04-30T06:43:22.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
104442025-04-30T06:43:22.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1a77fd6-a097-4b8b-bc36-fe416249323f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104452025-04-30T06:43:22.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
104462025-04-30T06:43:22.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae76b21d-4372-4928-ac19-a357bbafb26b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104472025-04-30T06:43:22.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
104482025-04-30T06:43:22.806ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
104492025-04-30T06:43:22.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0814298d-de0e-48ea-a235-e3f29f2f5e9b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104502025-04-30T06:43:22.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
104512025-04-30T06:43:22.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4250a985-e5f2-4af3-bb01-315823f47aa5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104522025-04-30T06:43:22.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
104532025-04-30T06:43:22.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20be8de6-3191-4ef1-a01e-8940055b1ccc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104542025-04-30T06:43:22.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
104552025-04-30T06:43:22.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98f6966d-8243-40ae-b63f-123b56d90f17", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104562025-04-30T06:43:22.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
104572025-04-30T06:43:22.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a0632cd-7f74-4699-91fc-36deb9065c03", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104582025-04-30T06:43:22.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
104592025-04-30T06:43:22.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40be9fa7-ba10-43f4-9cd5-0cc85d34c298", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104602025-04-30T06:43:22.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
104612025-04-30T06:43:22.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86ee0792-9ae9-444f-ba4d-d4e4f0f7f8d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104622025-04-30T06:43:22.883ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
104632025-04-30T06:43:22.883ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104642025-04-30T06:43:22.883ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104652025-04-30T06:43:22.887ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104662025-04-30T06:43:22.887ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104672025-04-30T06:43:22.889ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104682025-04-30T06:43:22.920ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104692025-04-30T06:43:22.931ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104702025-04-30T06:43:22.931ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104712025-04-30T06:43:22.931ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104722025-04-30T06:43:22.933ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104732025-04-30T06:43:22.938ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104742025-04-30T06:43:22.938ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
104752025-04-30T06:43:22.940ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104762025-04-30T06:43:22.940ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a01813be-fc6d-4f76-b657-e7e8f70ef779", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104772025-04-30T06:43:22.940ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
104782025-04-30T06:43:22.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "413e06f3-e7d1-4f04-a4a3-e6d3a80b6b60", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104792025-04-30T06:43:22.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
104802025-04-30T06:43:22.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c23f4194-95d3-4dd9-bd76-acd263a75599", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104812025-04-30T06:43:22.949ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
104822025-04-30T06:43:22.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24f6214-812d-4e90-9d45-222b153b78cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104832025-04-30T06:43:22.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
104842025-04-30T06:43:22.957ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "589051a0-1f36-47e7-a756-165a3b2dfc72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104852025-04-30T06:43:22.962ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
104862025-04-30T06:43:22.964ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ba89c1-9b9c-426f-9f7a-4470b73d9202", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104872025-04-30T06:43:22.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
104882025-04-30T06:43:22.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf08a37f-02ac-4946-85c2-d31536b69ce4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104892025-04-30T06:43:22.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
104902025-04-30T06:43:22.970ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "437685ef-0d32-4e41-9873-5d4a93c73775", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104912025-04-30T06:43:22.970ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
104922025-04-30T06:43:22.979ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104932025-04-30T06:43:22.980ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104942025-04-30T06:43:22.981ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
104952025-04-30T06:43:22.982ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
104962025-04-30T06:43:22.989ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0d9375d-400f-418d-bae0-7b915c8048e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
104972025-04-30T06:43:22.990ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
104982025-04-30T06:43:22.994ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
104992025-04-30T06:43:22.994ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
105002025-04-30T06:43:22.997ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f117624-5e05-41fc-8de2-3851303d1c85", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105012025-04-30T06:43:22.998ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
105022025-04-30T06:43:22.998ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
105032025-04-30T06:43:23.000ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
105042025-04-30T06:43:23.000ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105052025-04-30T06:43:23.003ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f1d731b-3dae-43df-bafb-057d9ac5bc09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105062025-04-30T06:43:23.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
105072025-04-30T06:43:23.008ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54528d72-6eb7-4b02-9f16-59e0e256479b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105082025-04-30T06:43:23.008ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
105092025-04-30T06:43:23.010ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f66e934-1d84-4d8b-a05b-6773e13b0af4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105102025-04-30T06:43:23.011ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
105112025-04-30T06:43:23.013ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb9feac8-262b-4be1-93a4-2da1e639db3e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105122025-04-30T06:43:23.013ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
105132025-04-30T06:43:23.015ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "019765c9-9086-4dee-9fc8-a0ca38a62b83", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105142025-04-30T06:43:23.015ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
105152025-04-30T06:43:23.017ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca020de3-c073-4062-a3d1-59b4654352ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105162025-04-30T06:43:23.018ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
105172025-04-30T06:43:23.018ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9645efa9-1c67-4978-aba8-c3e31db10664", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105182025-04-30T06:43:23.019ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
105192025-04-30T06:43:23.020ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b750bf5a-4b83-4928-b4c2-e83c1552c3f6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105202025-04-30T06:43:23.020ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
105212025-04-30T06:43:23.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fc1e687-4406-42c5-b6d7-34a0e44830a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105222025-04-30T06:43:23.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
105232025-04-30T06:43:23.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5761c12e-9c4b-4b6a-a456-87a5ceab335f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105242025-04-30T06:43:23.022ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105252025-04-30T06:43:23.023ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105262025-04-30T06:43:23.023ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105272025-04-30T06:43:23.023ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105282025-04-30T06:43:23.023ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105292025-04-30T06:43:23.038ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105302025-04-30T06:43:23.042ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105312025-04-30T06:43:23.042ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105322025-04-30T06:43:23.042ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105332025-04-30T06:43:23.042ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105342025-04-30T06:43:23.042ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105352025-04-30T06:43:23.046ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105362025-04-30T06:43:23.050ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105372025-04-30T06:43:23.050ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105382025-04-30T06:43:23.050ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105392025-04-30T06:43:23.050ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
105402025-04-30T06:43:23.051ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105412025-04-30T06:43:23.051ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105422025-04-30T06:43:23.055ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105432025-04-30T06:43:23.055ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105442025-04-30T06:43:23.055ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105452025-04-30T06:43:23.055ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105462025-04-30T06:43:23.057ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105472025-04-30T06:43:23.069ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105482025-04-30T06:43:23.069ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105492025-04-30T06:43:23.069ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
105502025-04-30T06:43:23.070ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105512025-04-30T06:43:23.075ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105522025-04-30T06:43:23.075ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
105532025-04-30T06:43:23.076ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105542025-04-30T06:43:23.161ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
105552025-04-30T06:43:23.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
105562025-04-30T06:43:23.168ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:61356 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105572025-04-30T06:43:23.170ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
105582025-04-30T06:43:23.170ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:61356 req_id = 2b11aade-36d6-40e4-b856-8d011870cd26 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105592025-04-30T06:43:23.171ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1039 local_addr = [::1]:45714 method = GET remote_addr = [::1]:61356 req_id = 2b11aade-36d6-40e4-b856-8d011870cd26 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105602025-04-30T06:43:23.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b11aade-36d6-40e4-b856-8d011870cd26", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:22 GMT"} })
105612025-04-30T06:43:23.174ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105622025-04-30T06:43:23.174ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105632025-04-30T06:43:23.175ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
105642025-04-30T06:43:23.176ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105652025-04-30T06:43:23.177ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105662025-04-30T06:43:23.177ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105672025-04-30T06:43:23.177ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105682025-04-30T06:43:23.178ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105692025-04-30T06:43:23.191ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105702025-04-30T06:43:23.200ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105712025-04-30T06:43:23.200ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105722025-04-30T06:43:23.200ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105732025-04-30T06:43:23.201ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105742025-04-30T06:43:23.351ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
105752025-04-30T06:43:23.359ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
105762025-04-30T06:43:23.359ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
105772025-04-30T06:43:23.359ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 941.394457ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
105782025-04-30T06:43:23.360ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
105792025-04-30T06:43:23.361ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105802025-04-30T06:43:23.361ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105812025-04-30T06:43:23.362ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105822025-04-30T06:43:23.362ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105832025-04-30T06:43:23.364ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105842025-04-30T06:43:23.382ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105852025-04-30T06:43:23.387ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105862025-04-30T06:43:23.388ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105872025-04-30T06:43:23.388ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105882025-04-30T06:43:23.389ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105892025-04-30T06:43:23.403ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105902025-04-30T06:43:23.411ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105912025-04-30T06:43:23.426ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105922025-04-30T06:43:23.426ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
105932025-04-30T06:43:23.427ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105942025-04-30T06:43:23.427ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
105952025-04-30T06:43:23.427ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
105962025-04-30T06:43:23.428ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 66 26 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 f&..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60087 req_id = e5bddb60-12f3-4174-8aa1-2e054170309c
105972025-04-30T06:43:23.432ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26150,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60087 req_id = e5bddb60-12f3-4174-8aa1-2e054170309c
105982025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
105992025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106002025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60087 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e5bddb60-12f3-4174-8aa1-2e054170309c
106012025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106022025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106032025-04-30T06:43:23.440ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106042025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
106052025-04-30T06:43:23.440ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106062025-04-30T06:43:23.447ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4268, tv_nsec: 240041482 } })
106072025-04-30T06:43:23.447ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
106082025-04-30T06:43:23.447ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 85 72 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .r..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63009 req_id = 1efc2a33-f8d3-422e-8b08-32e000e0ec5f
106092025-04-30T06:43:23.447ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34162,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63009 req_id = 1efc2a33-f8d3-422e-8b08-32e000e0ec5f
106102025-04-30T06:43:23.448ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106112025-04-30T06:43:23.448ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63009 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1efc2a33-f8d3-422e-8b08-32e000e0ec5f
106122025-04-30T06:43:23.448ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106132025-04-30T06:43:23.448ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
106142025-04-30T06:43:23.448ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106152025-04-30T06:43:23.556ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106162025-04-30T06:43:23.557ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 41 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Ao..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62473 req_id = f372378f-3129-4b78-b36b-a00c990b3eee
106172025-04-30T06:43:23.557ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16751,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62473 req_id = f372378f-3129-4b78-b36b-a00c990b3eee
106182025-04-30T06:43:23.557ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
106192025-04-30T06:43:23.557ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16751, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62473 req_id = f372378f-3129-4b78-b36b-a00c990b3eee
106202025-04-30T06:43:23.558ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
106212025-04-30T06:43:23.558ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
106222025-04-30T06:43:23.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
106232025-04-30T06:43:23.561ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9204f395-961c-49f9-a15c-9133ecdf57f8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106242025-04-30T06:43:23.561ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
106252025-04-30T06:43:23.563ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef923b92-3f88-4f6d-be98-5f08131ade59", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106262025-04-30T06:43:23.564ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
106272025-04-30T06:43:23.573ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4332086-a154-450f-8b10-1806f37502bd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106282025-04-30T06:43:23.573ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
106292025-04-30T06:43:23.585ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c00d8286-a3b0-40dc-971f-f00ef6943d82", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106302025-04-30T06:43:23.585ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
106312025-04-30T06:43:23.589ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad508e4-9c91-49d8-9d93-6f0f7c58a473", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106322025-04-30T06:43:23.590ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
106332025-04-30T06:43:23.596ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
106342025-04-30T06:43:23.596ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "163853ed-b3e6-4b2c-b224-b0a427607e46", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106352025-04-30T06:43:23.596ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
106362025-04-30T06:43:23.601ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d02ca5f7-d027-44ed-9ee3-8c89b0f91858", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106372025-04-30T06:43:23.603ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
106382025-04-30T06:43:23.610ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94151f23-7826-48a9-ae36-cb2853dc8ae0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106392025-04-30T06:43:23.610ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
106402025-04-30T06:43:23.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f272467-bad8-454a-b673-9281ab3848b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106412025-04-30T06:43:23.612ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
106422025-04-30T06:43:23.613ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "833c7a8a-8682-447a-85a4-817c470840ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106432025-04-30T06:43:23.614ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
106442025-04-30T06:43:23.616ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7297910-8427-4136-ba94-ca72d7bb26ef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106452025-04-30T06:43:23.616ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
106462025-04-30T06:43:23.617ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6462b9c0-a9c3-43d6-afa5-0cafe056319c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106472025-04-30T06:43:23.617ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
106482025-04-30T06:43:23.619ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d3c422-35a9-44df-9796-cceccf1bf00b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106492025-04-30T06:43:23.619ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
106502025-04-30T06:43:23.624ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b935e73-1ce8-4554-be52-c0f90f812b0e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106512025-04-30T06:43:23.624ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
106522025-04-30T06:43:23.635ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
106532025-04-30T06:43:23.638ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106542025-04-30T06:43:23.638ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 85a9cc09-281d-42d9-a8c3-49aa2852dac7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106552025-04-30T06:43:23.639ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 85a9cc09-281d-42d9-a8c3-49aa2852dac7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106562025-04-30T06:43:23.639ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 85a9cc09-281d-42d9-a8c3-49aa2852dac7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106572025-04-30T06:43:23.644ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 85a9cc09-281d-42d9-a8c3-49aa2852dac7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106582025-04-30T06:43:23.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa52eb6a-583f-48c0-978f-8efa0ca78b47", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106592025-04-30T06:43:23.646ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
106602025-04-30T06:43:23.647ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106612025-04-30T06:43:23.647ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106622025-04-30T06:43:23.647ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106632025-04-30T06:43:23.647ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
106642025-04-30T06:43:23.649ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106652025-04-30T06:43:23.650ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 11304 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 85a9cc09-281d-42d9-a8c3-49aa2852dac7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
106662025-04-30T06:43:23.650ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "85a9cc09-281d-42d9-a8c3-49aa2852dac7", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106672025-04-30T06:43:23.650ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106682025-04-30T06:43:23.651ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = bc462e76-771e-4fae-a7d2-0989dda72cf3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106692025-04-30T06:43:23.651ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = bc462e76-771e-4fae-a7d2-0989dda72cf3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106702025-04-30T06:43:23.651ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = bc462e76-771e-4fae-a7d2-0989dda72cf3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106712025-04-30T06:43:23.652ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = bc462e76-771e-4fae-a7d2-0989dda72cf3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106722025-04-30T06:43:23.653ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106732025-04-30T06:43:23.659ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106742025-04-30T06:43:23.659ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106752025-04-30T06:43:23.659ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
106762025-04-30T06:43:23.659ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106772025-04-30T06:43:23.661ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 9982 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = bc462e76-771e-4fae-a7d2-0989dda72cf3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
106782025-04-30T06:43:23.661ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc462e76-771e-4fae-a7d2-0989dda72cf3", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106792025-04-30T06:43:23.661ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106802025-04-30T06:43:23.661ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
106812025-04-30T06:43:23.662ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
106822025-04-30T06:43:23.662ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106832025-04-30T06:43:23.662ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
106842025-04-30T06:43:23.662ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106852025-04-30T06:43:23.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa107391-f7d6-4dc8-8f43-8b64d9afa2c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106862025-04-30T06:43:23.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
106872025-04-30T06:43:23.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d3fdb8f-18a5-4804-81d8-6a7b3fe1cca7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106882025-04-30T06:43:23.677ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
106892025-04-30T06:43:23.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96e0ef2f-ebf3-4739-9746-d68e507cab97", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106902025-04-30T06:43:23.680ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
106912025-04-30T06:43:23.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2b95e9-b6c2-48ed-b8c8-608fbb889437", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106922025-04-30T06:43:23.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
106932025-04-30T06:43:23.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6307d1a-e452-4b8b-bb9e-c68fcbc33a75", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106942025-04-30T06:43:23.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
106952025-04-30T06:43:23.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6b7b574-10f4-42c5-8b21-b8859afc703f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106962025-04-30T06:43:23.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
106972025-04-30T06:43:23.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a1c2f1c-1bfd-4904-8f77-7851e120afd9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
106982025-04-30T06:43:23.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
106992025-04-30T06:43:23.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0fa337f-2875-4565-a037-d3616947ad28", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107002025-04-30T06:43:23.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
107012025-04-30T06:43:23.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a49884d-0a24-48aa-be3a-207480a31aa1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107022025-04-30T06:43:23.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
107032025-04-30T06:43:23.705ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107042025-04-30T06:43:23.705ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107052025-04-30T06:43:23.707ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
107062025-04-30T06:43:23.708ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107072025-04-30T06:43:23.709ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107082025-04-30T06:43:23.710ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107092025-04-30T06:43:23.710ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107102025-04-30T06:43:23.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d707a0-5e51-4805-a39d-8407ff0103ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107112025-04-30T06:43:23.731ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
107122025-04-30T06:43:23.743ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcdbe02e-b952-4129-949b-942386ce2fdb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107132025-04-30T06:43:23.744ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
107142025-04-30T06:43:23.750ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e22634b-b414-4f94-8789-028e1dc10535", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107152025-04-30T06:43:23.750ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
107162025-04-30T06:43:23.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25aeb07f-5369-4378-afda-d93cd903fd40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107172025-04-30T06:43:23.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
107182025-04-30T06:43:23.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0074f9f-15ab-4bf8-9fd0-0af62478f4dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107192025-04-30T06:43:23.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
107202025-04-30T06:43:23.758ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107212025-04-30T06:43:23.759ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107222025-04-30T06:43:23.759ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107232025-04-30T06:43:23.759ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
107242025-04-30T06:43:23.759ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107252025-04-30T06:43:23.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24e1584d-78c6-4c68-8796-395f7ba34d19", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107262025-04-30T06:43:23.760ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
107272025-04-30T06:43:23.763ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107282025-04-30T06:43:23.763ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107292025-04-30T06:43:23.765ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = bbaf36b5-9f70-4d81-a8ab-a6d27174ba0e uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
107302025-04-30T06:43:23.769ZINFOtest_snapshot_unwind (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4213 local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = bbaf36b5-9f70-4d81-a8ab-a6d27174ba0e response_code = 200 uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
107312025-04-30T06:43:23.769ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107322025-04-30T06:43:23.777ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107332025-04-30T06:43:23.777ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107342025-04-30T06:43:23.777ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
107352025-04-30T06:43:23.778ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107362025-04-30T06:43:23.778ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107372025-04-30T06:43:23.790ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107382025-04-30T06:43:23.790ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107392025-04-30T06:43:23.790ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
107402025-04-30T06:43:23.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83786430-1ba4-4bd1-90ed-dda87e4e0091", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107412025-04-30T06:43:23.792ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
107422025-04-30T06:43:23.797ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107432025-04-30T06:43:23.797ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
107442025-04-30T06:43:23.797ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107452025-04-30T06:43:23.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3c7489-c6f9-4454-9dd4-3aea4d087f65", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107462025-04-30T06:43:23.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
107472025-04-30T06:43:23.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e758ef1e-faae-440d-8cbf-82c6afed5063", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107482025-04-30T06:43:23.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
107492025-04-30T06:43:23.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a618ca0-8a5d-4021-a3ff-820deedfcda7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107502025-04-30T06:43:23.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
107512025-04-30T06:43:23.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca4307a-4f1c-4329-b8a2-766fcf953061", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107522025-04-30T06:43:23.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
107532025-04-30T06:43:23.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22da1461-111a-46a8-8233-4e7e02ebda85", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107542025-04-30T06:43:23.835ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
107552025-04-30T06:43:23.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "113b9120-827a-499d-9930-c53ad9d5325a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107562025-04-30T06:43:23.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
107572025-04-30T06:43:23.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2669539-5825-4df6-b2e4-bc9bfaa253d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107582025-04-30T06:43:23.842ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
107592025-04-30T06:43:23.851ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
107602025-04-30T06:43:23.853ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "360cbc90-c551-460f-8671-18bdf123eaf8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107612025-04-30T06:43:23.853ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
107622025-04-30T06:43:23.855ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9904a46b-0a4f-4f4c-8e93-f1f44a367169", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107632025-04-30T06:43:23.855ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
107642025-04-30T06:43:23.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4336c379-c6f1-427b-9565-7d62b00a1cf7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107652025-04-30T06:43:23.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
107662025-04-30T06:43:23.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0227cac8-1414-4d61-8df8-c9286dc7f6e5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107672025-04-30T06:43:23.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
107682025-04-30T06:43:23.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e30877f-b20b-4c2f-bd84-18d7ffee3cdb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107692025-04-30T06:43:23.860ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
107702025-04-30T06:43:23.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "586d2b8d-9655-4739-bb45-cc56fc79556a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107712025-04-30T06:43:23.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
107722025-04-30T06:43:23.862ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7c3d9c0-5e49-4895-a68b-8d24b2eb8226", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
107732025-04-30T06:43:23.862ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107742025-04-30T06:43:23.863ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107752025-04-30T06:43:23.863ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107762025-04-30T06:43:23.863ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107772025-04-30T06:43:23.863ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107782025-04-30T06:43:23.868ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107792025-04-30T06:43:23.872ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107802025-04-30T06:43:23.872ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107812025-04-30T06:43:23.872ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107822025-04-30T06:43:23.872ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107832025-04-30T06:43:23.872ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107842025-04-30T06:43:23.873ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107852025-04-30T06:43:23.880ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107862025-04-30T06:43:23.888ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107872025-04-30T06:43:23.889ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107882025-04-30T06:43:23.889ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107892025-04-30T06:43:23.890ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107902025-04-30T06:43:23.973ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
107912025-04-30T06:43:23.973ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
107922025-04-30T06:43:23.973ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107932025-04-30T06:43:23.974ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107942025-04-30T06:43:23.974ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107952025-04-30T06:43:23.974ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
107962025-04-30T06:43:23.975ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107972025-04-30T06:43:23.975ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:63626 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107982025-04-30T06:43:23.976ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:63626 req_id = 00b09770-fefb-4ca1-b323-9c51ffe9f1af sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
107992025-04-30T06:43:23.976ZDEBGtest_snapshot_unwind (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108002025-04-30T06:43:23.976ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108012025-04-30T06:43:23.976ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 600 local_addr = [::1]:45714 method = GET remote_addr = [::1]:63626 req_id = 00b09770-fefb-4ca1-b323-9c51ffe9f1af response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108022025-04-30T06:43:23.979ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
108032025-04-30T06:43:23.985ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
108042025-04-30T06:43:23.986ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b09770-fefb-4ca1-b323-9c51ffe9f1af", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:23 GMT"} })
108052025-04-30T06:43:23.988ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108062025-04-30T06:43:23.988ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108072025-04-30T06:43:23.988ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
108082025-04-30T06:43:23.990ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108092025-04-30T06:43:23.990ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108102025-04-30T06:43:23.990ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108112025-04-30T06:43:23.990ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108122025-04-30T06:43:23.991ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108132025-04-30T06:43:23.992ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108142025-04-30T06:43:23.997ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108152025-04-30T06:43:23.997ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108162025-04-30T06:43:23.997ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
108172025-04-30T06:43:23.997ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108182025-04-30T06:43:24.003ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 12 table_name = fields_string
108192025-04-30T06:43:24.003ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
108202025-04-30T06:43:24.005ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108212025-04-30T06:43:24.005ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
108222025-04-30T06:43:24.006ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108232025-04-30T06:43:24.007ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108242025-04-30T06:43:24.012ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108252025-04-30T06:43:24.012ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108262025-04-30T06:43:24.012ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108272025-04-30T06:43:24.012ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108282025-04-30T06:43:24.013ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = fields_u16
108292025-04-30T06:43:24.013ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
108302025-04-30T06:43:24.017ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = fields_uuid
108312025-04-30T06:43:24.017ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
108322025-04-30T06:43:24.023ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = measurements_histogramu64
108332025-04-30T06:43:24.023ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
108342025-04-30T06:43:24.030ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
108352025-04-30T06:43:24.036ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108362025-04-30T06:43:24.080ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
108372025-04-30T06:43:24.080ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
108382025-04-30T06:43:24.086ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
108392025-04-30T06:43:24.086ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
108402025-04-30T06:43:24.087ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 726.087943ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
108412025-04-30T06:43:24.087ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
108422025-04-30T06:43:24.087ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108432025-04-30T06:43:24.087ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108442025-04-30T06:43:24.087ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108452025-04-30T06:43:24.087ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108462025-04-30T06:43:24.088ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108472025-04-30T06:43:24.092ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108482025-04-30T06:43:24.095ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108492025-04-30T06:43:24.095ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108502025-04-30T06:43:24.095ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108512025-04-30T06:43:24.096ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108522025-04-30T06:43:24.101ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
108532025-04-30T06:43:24.101ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
108542025-04-30T06:43:24.101ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
108552025-04-30T06:43:24.102ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
108562025-04-30T06:43:24.102ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
108572025-04-30T06:43:24.102ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
108582025-04-30T06:43:24.102ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
108592025-04-30T06:43:24.103ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
108602025-04-30T06:43:24.103ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
108612025-04-30T06:43:24.103ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
108622025-04-30T06:43:24.103ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
108632025-04-30T06:43:24.106ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108642025-04-30T06:43:24.109ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108652025-04-30T06:43:24.115ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 12.271791ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
108662025-04-30T06:43:24.116ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
108672025-04-30T06:43:24.116ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 151.773\u{b5}s iteration = 3 status = {"error":"task disabled"}
108682025-04-30T06:43:24.116ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
108692025-04-30T06:43:24.116ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
108702025-04-30T06:43:24.116ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108712025-04-30T06:43:24.116ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108722025-04-30T06:43:24.117ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108732025-04-30T06:43:24.117ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108742025-04-30T06:43:24.117ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
108752025-04-30T06:43:24.117ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
108762025-04-30T06:43:24.117ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108772025-04-30T06:43:24.118ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
108782025-04-30T06:43:24.118ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108792025-04-30T06:43:24.119ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108802025-04-30T06:43:24.121ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
108812025-04-30T06:43:24.121ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
108822025-04-30T06:43:24.121ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108832025-04-30T06:43:24.121ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108842025-04-30T06:43:24.121ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
108852025-04-30T06:43:24.121ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
108862025-04-30T06:43:24.121ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108872025-04-30T06:43:24.121ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
108882025-04-30T06:43:24.121ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108892025-04-30T06:43:24.122ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108902025-04-30T06:43:24.122ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
108912025-04-30T06:43:24.122ZINFOtest_snapshot_unwind (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
108922025-04-30T06:43:24.122ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = instance_updater elapsed = 75.53\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
108932025-04-30T06:43:24.122ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
108942025-04-30T06:43:24.122ZINFOtest_snapshot_unwind (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
108952025-04-30T06:43:24.122ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108962025-04-30T06:43:24.122ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108972025-04-30T06:43:24.122ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108982025-04-30T06:43:24.122ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108992025-04-30T06:43:24.128ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109002025-04-30T06:43:24.129ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 67 b3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 g............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51827 req_id = 5fc802ff-7fb9-436e-bc4b-70f325a8b565
109012025-04-30T06:43:24.129ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26547,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51827 req_id = 5fc802ff-7fb9-436e-bc4b-70f325a8b565
109022025-04-30T06:43:24.129ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
109032025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109042025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51827 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5fc802ff-7fb9-436e-bc4b-70f325a8b565
109052025-04-30T06:43:24.130ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 18 d7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53475 req_id = fda70fea-3492-4fdf-bf5c-539ee6c59eea
109062025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6359,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53475 req_id = fda70fea-3492-4fdf-bf5c-539ee6c59eea
109072025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
109082025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109092025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53475 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fda70fea-3492-4fdf-bf5c-539ee6c59eea
109102025-04-30T06:43:24.130ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 50 94 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 P............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61064 req_id = adbc4458-d739-4995-871d-6b11db9e1b98
109112025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20628,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61064 req_id = adbc4458-d739-4995-871d-6b11db9e1b98
109122025-04-30T06:43:24.130ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
109132025-04-30T06:43:24.131ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109142025-04-30T06:43:24.131ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61064 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = adbc4458-d739-4995-871d-6b11db9e1b98
109152025-04-30T06:43:24.132ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109162025-04-30T06:43:24.133ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109172025-04-30T06:43:24.133ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109182025-04-30T06:43:24.133ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
109192025-04-30T06:43:24.133ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109202025-04-30T06:43:24.133ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4268, tv_nsec: 932425952 } })
109212025-04-30T06:43:24.133ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
109222025-04-30T06:43:24.133ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4268, tv_nsec: 932825689 } })
109232025-04-30T06:43:24.133ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
109242025-04-30T06:43:24.134ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4268, tv_nsec: 933074715 } })
109252025-04-30T06:43:24.134ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
109262025-04-30T06:43:24.134ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109272025-04-30T06:43:24.144ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109282025-04-30T06:43:24.144ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109292025-04-30T06:43:24.144ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109302025-04-30T06:43:24.145ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109312025-04-30T06:43:24.145ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109322025-04-30T06:43:24.164ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109332025-04-30T06:43:24.164ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109342025-04-30T06:43:24.164ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109352025-04-30T06:43:24.165ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109362025-04-30T06:43:24.167ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109372025-04-30T06:43:24.167ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109382025-04-30T06:43:24.168ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109392025-04-30T06:43:24.169ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109402025-04-30T06:43:24.179ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
109412025-04-30T06:43:24.179ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109422025-04-30T06:43:24.180ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109432025-04-30T06:43:24.180ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109442025-04-30T06:43:24.181ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109452025-04-30T06:43:24.186ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109462025-04-30T06:43:24.186ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
109472025-04-30T06:43:24.186ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109482025-04-30T06:43:24.186ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109492025-04-30T06:43:24.186ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109502025-04-30T06:43:24.187ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109512025-04-30T06:43:24.188ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: dc 56 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .V..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53414 req_id = 9180e9bf-19fc-4869-b191-52e71535564f
109522025-04-30T06:43:24.188ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56406,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53414 req_id = 9180e9bf-19fc-4869-b191-52e71535564f
109532025-04-30T06:43:24.189ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109542025-04-30T06:43:24.189ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53414 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9180e9bf-19fc-4869-b191-52e71535564f
109552025-04-30T06:43:24.189ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 c8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62738 req_id = bc76577b-5306-47ba-9cf8-73434c7aa2a8
109562025-04-30T06:43:24.189ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26056,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62738 req_id = bc76577b-5306-47ba-9cf8-73434c7aa2a8
109572025-04-30T06:43:24.189ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109582025-04-30T06:43:24.190ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62738 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bc76577b-5306-47ba-9cf8-73434c7aa2a8
109592025-04-30T06:43:24.190ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 41 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .A..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58049 req_id = f5fa39bd-8b86-4d95-8110-217946b3740c
109602025-04-30T06:43:24.190ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41537,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58049 req_id = f5fa39bd-8b86-4d95-8110-217946b3740c
109612025-04-30T06:43:24.190ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109622025-04-30T06:43:24.190ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58049 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f5fa39bd-8b86-4d95-8110-217946b3740c
109632025-04-30T06:43:24.273ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
109642025-04-30T06:43:24.274ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
109652025-04-30T06:43:24.354ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
109662025-04-30T06:43:24.354ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
109672025-04-30T06:43:24.354ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109682025-04-30T06:43:24.354ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109692025-04-30T06:43:24.359ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
109702025-04-30T06:43:24.360ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
109712025-04-30T06:43:24.443ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
109722025-04-30T06:43:24.443ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
109732025-04-30T06:43:24.450ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
109742025-04-30T06:43:24.451ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
109752025-04-30T06:43:24.452ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
109762025-04-30T06:43:24.461ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109772025-04-30T06:43:24.470ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109782025-04-30T06:43:24.470ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109792025-04-30T06:43:24.470ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
109802025-04-30T06:43:24.471ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109812025-04-30T06:43:24.474ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109822025-04-30T06:43:24.474ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109832025-04-30T06:43:24.475ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109842025-04-30T06:43:24.478ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109852025-04-30T06:43:24.480ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f7 88 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49332 req_id = 93ecf56d-9465-4f07-b109-3016f19e99bc
109862025-04-30T06:43:24.480ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63368,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49332 req_id = 93ecf56d-9465-4f07-b109-3016f19e99bc
109872025-04-30T06:43:24.481ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
109882025-04-30T06:43:24.481ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63368, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49332 req_id = 93ecf56d-9465-4f07-b109-3016f19e99bc
109892025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
109902025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
109912025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
109922025-04-30T06:43:24.484ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: aa 7c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .|..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59636 req_id = 6ed64fae-c25d-4a2b-97cb-e12fd08ffdd7
109932025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43644,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59636 req_id = 6ed64fae-c25d-4a2b-97cb-e12fd08ffdd7
109942025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
109952025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109962025-04-30T06:43:24.484ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59636 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6ed64fae-c25d-4a2b-97cb-e12fd08ffdd7
109972025-04-30T06:43:24.486ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109982025-04-30T06:43:24.489ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109992025-04-30T06:43:24.490ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110002025-04-30T06:43:24.490ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110012025-04-30T06:43:24.490ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110022025-04-30T06:43:24.490ZINFOtest_snapshot_unwind (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
110032025-04-30T06:43:24.490ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 368.825601ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
110042025-04-30T06:43:24.491ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
110052025-04-30T06:43:24.491ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110062025-04-30T06:43:24.491ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110072025-04-30T06:43:24.491ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110082025-04-30T06:43:24.492ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110092025-04-30T06:43:24.518ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4269, tv_nsec: 317079730 } })
110102025-04-30T06:43:24.521ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
110112025-04-30T06:43:24.568ZINFOtest_snapshot_unwind (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
110122025-04-30T06:43:24.568ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:45714/vpc-routes
110132025-04-30T06:43:24.568ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110142025-04-30T06:43:24.577ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110152025-04-30T06:43:24.577ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
110162025-04-30T06:43:24.577ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110172025-04-30T06:43:24.577ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110182025-04-30T06:43:24.577ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110192025-04-30T06:43:24.577ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110202025-04-30T06:43:24.579ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110212025-04-30T06:43:24.579ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
110222025-04-30T06:43:24.580ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110232025-04-30T06:43:24.589ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40767 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110242025-04-30T06:43:24.606ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110252025-04-30T06:43:24.611ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110262025-04-30T06:43:24.611ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110272025-04-30T06:43:24.611ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110282025-04-30T06:43:24.613ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110292025-04-30T06:43:24.620ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40767 req_id = 988976c3-cf2f-4af4-99e1-be82c553a449 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110302025-04-30T06:43:24.620ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 23 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e #...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64914 req_id = dec45f89-a5a4-44f8-bab5-01d655da9b69
110312025-04-30T06:43:24.622ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8983,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64914 req_id = dec45f89-a5a4-44f8-bab5-01d655da9b69
110322025-04-30T06:43:24.622ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110332025-04-30T06:43:24.622ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64914 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dec45f89-a5a4-44f8-bab5-01d655da9b69
110342025-04-30T06:43:24.623ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 2900 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40767 req_id = 988976c3-cf2f-4af4-99e1-be82c553a449 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
110352025-04-30T06:43:24.691ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:45714/v2p
110362025-04-30T06:43:24.692ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:62231 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110372025-04-30T06:43:24.693ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
110382025-04-30T06:43:24.756ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
110392025-04-30T06:43:24.756ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
110402025-04-30T06:43:24.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45714/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "988976c3-cf2f-4af4-99e1-be82c553a449", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110412025-04-30T06:43:24.764ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110422025-04-30T06:43:24.764ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110432025-04-30T06:43:24.765ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110442025-04-30T06:43:24.767ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:62231 req_id = b97ff431-e8bf-4d7f-b80a-736c67e9f296 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110452025-04-30T06:43:24.767ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110462025-04-30T06:43:24.767ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110472025-04-30T06:43:24.769ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1070 local_addr = [::1]:45714 method = GET remote_addr = [::1]:62231 req_id = b97ff431-e8bf-4d7f-b80a-736c67e9f296 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
110482025-04-30T06:43:24.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:45714/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97ff431-e8bf-4d7f-b80a-736c67e9f296", "content-length": "2", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110492025-04-30T06:43:24.780ZINFOtest_snapshot_unwind (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
110502025-04-30T06:43:24.781ZINFOtest_snapshot_unwind (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
110512025-04-30T06:43:24.781ZINFOtest_snapshot_unwind (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
110522025-04-30T06:43:24.782ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 661.658311ms iteration = 2 status = {}
110532025-04-30T06:43:24.785ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0805e11c-98f7-4a6c-afbe-f0547ed168e2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110542025-04-30T06:43:24.785ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
110552025-04-30T06:43:24.786ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110562025-04-30T06:43:24.787ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110572025-04-30T06:43:24.787ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110582025-04-30T06:43:24.787ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
110592025-04-30T06:43:24.802ZINFOtest_snapshot_unwind (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
110602025-04-30T06:43:24.802ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:45714/eip-gateways
110612025-04-30T06:43:24.805ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = PUT remote_addr = [::1]:40767 req_id = 139bf8c5-7ba0-47ba-aec5-ddac1d299298 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110622025-04-30T06:43:24.806ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 542 local_addr = [::1]:45714 method = PUT remote_addr = [::1]:40767 req_id = 139bf8c5-7ba0-47ba-aec5-ddac1d299298 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110632025-04-30T06:43:24.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45714/eip-gateways", status: 204, headers: {"x-request-id": "139bf8c5-7ba0-47ba-aec5-ddac1d299298", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110642025-04-30T06:43:24.812ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 689.806123ms iteration = 2 status = {}
110652025-04-30T06:43:24.829ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110662025-04-30T06:43:24.829ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110672025-04-30T06:43:24.831ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110682025-04-30T06:43:24.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bead198-3848-406b-829a-309a445f6bc8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110692025-04-30T06:43:24.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
110702025-04-30T06:43:24.845ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
110712025-04-30T06:43:24.845ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110722025-04-30T06:43:24.845ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110732025-04-30T06:43:24.845ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110742025-04-30T06:43:24.846ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110752025-04-30T06:43:24.866ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110762025-04-30T06:43:24.872ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
110772025-04-30T06:43:24.872ZERROtest_snapshot_unwind (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
110782025-04-30T06:43:24.872ZERROtest_snapshot_unwind (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
110792025-04-30T06:43:24.872ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 756.127936ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
110802025-04-30T06:43:24.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50066aee-5d64-41f5-8cfe-1fc4c3c42f1f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110812025-04-30T06:43:24.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
110822025-04-30T06:43:24.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b631c5-3cc7-4a81-b259-21fe237914f8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110832025-04-30T06:43:24.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
110842025-04-30T06:43:24.882ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccd606bf-48b8-4c26-9957-9dd179afa9ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110852025-04-30T06:43:24.883ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
110862025-04-30T06:43:24.883ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
110872025-04-30T06:43:24.885ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "922c1c35-14ee-4dff-ba02-4bbd816f62ad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110882025-04-30T06:43:24.885ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
110892025-04-30T06:43:24.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14dc19fe-24ea-4f9b-82f0-d4426f1a3cde", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110902025-04-30T06:43:24.889ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
110912025-04-30T06:43:24.901ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2cbf658-961b-46ac-94d7-3e29fcdd704a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110922025-04-30T06:43:24.901ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
110932025-04-30T06:43:24.902ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bfdc13f-d424-42f1-99cd-d2719ba8cafd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110942025-04-30T06:43:24.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
110952025-04-30T06:43:24.904ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffa76c2-078c-4482-9115-c9c788edcb8a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110962025-04-30T06:43:24.904ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
110972025-04-30T06:43:24.905ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a19253-b8d3-441c-8d63-2278d5c86f52", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
110982025-04-30T06:43:24.905ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
110992025-04-30T06:43:24.946ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111002025-04-30T06:43:24.947ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111012025-04-30T06:43:24.947ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111022025-04-30T06:43:24.947ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111032025-04-30T06:43:24.965ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111042025-04-30T06:43:25.005ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111052025-04-30T06:43:25.005ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111062025-04-30T06:43:25.005ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
111072025-04-30T06:43:25.006ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
111082025-04-30T06:43:25.007ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0759422-7fb8-4e76-941f-334638ad56fe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111092025-04-30T06:43:25.007ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
111102025-04-30T06:43:25.008ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111112025-04-30T06:43:25.028ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111122025-04-30T06:43:25.028ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111132025-04-30T06:43:25.028ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111142025-04-30T06:43:25.029ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111152025-04-30T06:43:25.033ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
111162025-04-30T06:43:25.033ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111172025-04-30T06:43:25.036ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111182025-04-30T06:43:25.037ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111192025-04-30T06:43:25.037ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111202025-04-30T06:43:25.038ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
111212025-04-30T06:43:25.038ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111222025-04-30T06:43:25.039ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba6f64d8-183d-4950-ae32-dab379b95608", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111232025-04-30T06:43:25.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
111242025-04-30T06:43:25.048ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7a8c89c-db9f-437d-aed6-bfecc73e47e0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111252025-04-30T06:43:25.048ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
111262025-04-30T06:43:25.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0df13693-884e-447e-b035-06059b3bd2d7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111272025-04-30T06:43:25.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
111282025-04-30T06:43:25.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22fba083-17e6-4f4e-bc04-23546be95aa3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111292025-04-30T06:43:25.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
111302025-04-30T06:43:25.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99650758-eda0-4c3c-b44e-81e4a9a17adc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111312025-04-30T06:43:25.088ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
111322025-04-30T06:43:25.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab54135-5441-46c8-951a-c0ddbb1b45d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111332025-04-30T06:43:25.091ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
111342025-04-30T06:43:25.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d144dd55-2812-45dc-9732-3898dfed6fcd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111352025-04-30T06:43:25.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
111362025-04-30T06:43:25.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ddad63f-f5e5-45a8-8cb7-a6733d13745c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111372025-04-30T06:43:25.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
111382025-04-30T06:43:25.097ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e85895d-7b9d-4322-8a95-13c8d60dfa72", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111392025-04-30T06:43:25.097ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
111402025-04-30T06:43:25.097ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
111412025-04-30T06:43:25.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "663cbfce-eb62-435d-83a7-f33066e85ad4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111422025-04-30T06:43:25.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
111432025-04-30T06:43:25.100ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da9e0d79-0f00-4652-a896-0b3223547206", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111442025-04-30T06:43:25.100ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
111452025-04-30T06:43:25.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7565e429-78da-412c-b046-3cc94f51bad3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111462025-04-30T06:43:25.102ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
111472025-04-30T06:43:25.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5a80de8-6008-44b0-9ef2-ef3a7d54657b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111482025-04-30T06:43:25.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
111492025-04-30T06:43:25.106ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
111502025-04-30T06:43:25.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "679d0253-c09b-44c3-a816-cad26a13609b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111512025-04-30T06:43:25.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
111522025-04-30T06:43:25.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc44b05c-b02b-4957-801d-93278ab11c14", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111532025-04-30T06:43:25.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
111542025-04-30T06:43:25.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89bfeb05-b5b3-47fe-8a96-f7364be53e80", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111552025-04-30T06:43:25.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
111562025-04-30T06:43:25.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f299b5fa-8ef0-4483-a6e2-4b81442f7a3b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111572025-04-30T06:43:25.119ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
111582025-04-30T06:43:25.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1855de70-0b94-4548-a321-c60b8200f282", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111592025-04-30T06:43:25.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
111602025-04-30T06:43:25.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae6ebf04-9b18-47ff-a93b-5e1b68f26ba9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111612025-04-30T06:43:25.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
111622025-04-30T06:43:25.125ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3be5010-2aa2-412e-95ad-e11e77202857", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111632025-04-30T06:43:25.125ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
111642025-04-30T06:43:25.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56569bc-07bd-42ee-9fba-a73014980980", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111652025-04-30T06:43:25.132ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
111662025-04-30T06:43:25.132ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59a325db-701b-4c53-aa2f-bffc769c6c25", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111672025-04-30T06:43:25.132ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
111682025-04-30T06:43:25.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b2dd2b5-69d0-4f53-b7e0-fa179b877918", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111692025-04-30T06:43:25.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
111702025-04-30T06:43:25.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a3ffc9f-c415-4f54-89cb-30d6a3b5a6db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111712025-04-30T06:43:25.139ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
111722025-04-30T06:43:25.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09aed176-6f6f-4a72-be45-999b4d5a7169", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111732025-04-30T06:43:25.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
111742025-04-30T06:43:25.150ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111752025-04-30T06:43:25.150ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111762025-04-30T06:43:25.150ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111772025-04-30T06:43:25.150ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111782025-04-30T06:43:25.151ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111792025-04-30T06:43:25.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0971654c-acf4-4a75-a66f-3cb1b4e225f7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111802025-04-30T06:43:25.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
111812025-04-30T06:43:25.154ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111822025-04-30T06:43:25.162ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111832025-04-30T06:43:25.162ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111842025-04-30T06:43:25.162ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111852025-04-30T06:43:25.164ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111862025-04-30T06:43:25.165ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111872025-04-30T06:43:25.165ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
111882025-04-30T06:43:25.166ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111892025-04-30T06:43:25.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b41c72de-44dc-4fee-857b-ec8ea0b0840b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111902025-04-30T06:43:25.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
111912025-04-30T06:43:25.175ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0780742-d5fa-496a-a52a-c5b21dc4efb9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111922025-04-30T06:43:25.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
111932025-04-30T06:43:25.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7faccf2-f0bb-4821-894b-ebba1a0398cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111942025-04-30T06:43:25.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
111952025-04-30T06:43:25.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f00a7df-e7fb-4d24-83f4-811f94b4ca99", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111962025-04-30T06:43:25.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
111972025-04-30T06:43:25.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23807571-b1a1-4ceb-a107-62f3e41b4c12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
111982025-04-30T06:43:25.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
111992025-04-30T06:43:25.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4d009a4-dac9-4eb9-8b92-75f9fda578a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
112002025-04-30T06:43:25.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
112012025-04-30T06:43:25.231ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93b4b89f-4508-41e2-b961-98c2abbb9610", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
112022025-04-30T06:43:25.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112032025-04-30T06:43:25.231ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112042025-04-30T06:43:25.231ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112052025-04-30T06:43:25.231ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112062025-04-30T06:43:25.232ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112072025-04-30T06:43:25.238ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112082025-04-30T06:43:25.243ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112092025-04-30T06:43:25.243ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112102025-04-30T06:43:25.243ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112112025-04-30T06:43:25.243ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112122025-04-30T06:43:25.243ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112132025-04-30T06:43:25.245ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112142025-04-30T06:43:25.245ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
112152025-04-30T06:43:25.247ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112162025-04-30T06:43:25.252ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112172025-04-30T06:43:25.252ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112182025-04-30T06:43:25.252ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112192025-04-30T06:43:25.253ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112202025-04-30T06:43:25.316ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
112212025-04-30T06:43:25.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
112222025-04-30T06:43:25.317ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112232025-04-30T06:43:25.317ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112242025-04-30T06:43:25.317ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112252025-04-30T06:43:25.317ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
112262025-04-30T06:43:25.318ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112272025-04-30T06:43:25.319ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:39195 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112282025-04-30T06:43:25.320ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:39195 req_id = 748ec229-dea5-44d7-b346-dd2013d5e2f5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112292025-04-30T06:43:25.323ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 2990 local_addr = [::1]:45714 method = GET remote_addr = [::1]:39195 req_id = 748ec229-dea5-44d7-b346-dd2013d5e2f5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112302025-04-30T06:43:25.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "748ec229-dea5-44d7-b346-dd2013d5e2f5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:24 GMT"} })
112312025-04-30T06:43:25.327ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112322025-04-30T06:43:25.327ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112332025-04-30T06:43:25.327ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
112342025-04-30T06:43:25.334ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112352025-04-30T06:43:25.334ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112362025-04-30T06:43:25.334ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112372025-04-30T06:43:25.334ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112382025-04-30T06:43:25.335ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112392025-04-30T06:43:25.336ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112402025-04-30T06:43:25.345ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112412025-04-30T06:43:25.345ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112422025-04-30T06:43:25.345ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
112432025-04-30T06:43:25.345ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112442025-04-30T06:43:25.361ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112452025-04-30T06:43:25.369ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112462025-04-30T06:43:25.370ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112472025-04-30T06:43:25.370ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112482025-04-30T06:43:25.371ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112492025-04-30T06:43:25.378ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112502025-04-30T06:43:25.378ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
112512025-04-30T06:43:25.378ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112522025-04-30T06:43:25.438ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
112532025-04-30T06:43:25.438ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
112542025-04-30T06:43:25.438ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.351593682s iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
112552025-04-30T06:43:25.438ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
112562025-04-30T06:43:25.438ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112572025-04-30T06:43:25.438ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112582025-04-30T06:43:25.438ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112592025-04-30T06:43:25.438ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112602025-04-30T06:43:25.439ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112612025-04-30T06:43:25.441ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112622025-04-30T06:43:25.446ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112632025-04-30T06:43:25.446ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112642025-04-30T06:43:25.446ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112652025-04-30T06:43:25.446ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112662025-04-30T06:43:25.461ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
112672025-04-30T06:43:25.463ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112682025-04-30T06:43:25.466ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
112692025-04-30T06:43:25.471ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b9 7d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .}..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63547 req_id = a42514a6-3d4e-4be8-b44c-615eac4c03d6
112702025-04-30T06:43:25.471ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47485,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63547 req_id = a42514a6-3d4e-4be8-b44c-615eac4c03d6
112712025-04-30T06:43:25.471ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
112722025-04-30T06:43:25.472ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112732025-04-30T06:43:25.472ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63547 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a42514a6-3d4e-4be8-b44c-615eac4c03d6
112742025-04-30T06:43:25.472ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4270, tv_nsec: 271724117 } })
112752025-04-30T06:43:25.472ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
112762025-04-30T06:43:25.473ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ef 23 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .#..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54170 req_id = 26c84158-8199-4cec-a5f7-b01eea245d86
112772025-04-30T06:43:25.473ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61219,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54170 req_id = 26c84158-8199-4cec-a5f7-b01eea245d86
112782025-04-30T06:43:25.473ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112792025-04-30T06:43:25.473ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54170 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 26c84158-8199-4cec-a5f7-b01eea245d86
112802025-04-30T06:43:25.571ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112812025-04-30T06:43:25.571ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112822025-04-30T06:43:25.572ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112832025-04-30T06:43:25.572ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112842025-04-30T06:43:25.572ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
112852025-04-30T06:43:25.572ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112862025-04-30T06:43:25.573ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cc 7c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .|..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64829 req_id = bbac4a4f-3885-4da8-9bb9-1ee84df0f4cb
112872025-04-30T06:43:25.573ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52348,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64829 req_id = bbac4a4f-3885-4da8-9bb9-1ee84df0f4cb
112882025-04-30T06:43:25.573ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
112892025-04-30T06:43:25.573ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52348, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64829 req_id = bbac4a4f-3885-4da8-9bb9-1ee84df0f4cb
112902025-04-30T06:43:25.573ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
112912025-04-30T06:43:25.573ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
112922025-04-30T06:43:25.574ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
112932025-04-30T06:43:25.576ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112942025-04-30T06:43:25.582ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112952025-04-30T06:43:25.582ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112962025-04-30T06:43:25.582ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
112972025-04-30T06:43:25.583ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112982025-04-30T06:43:25.614ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e48192c-7a92-4a03-bc37-19548cf2468f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
112992025-04-30T06:43:25.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
113002025-04-30T06:43:25.615ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113012025-04-30T06:43:25.616ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
113022025-04-30T06:43:25.630ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113032025-04-30T06:43:25.640ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
113042025-04-30T06:43:25.641ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113052025-04-30T06:43:25.643ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 837c22cf-bcb8-420d-8b3a-390c0efda3dc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113062025-04-30T06:43:25.643ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb964d8-f52f-419e-be4f-ee3d09afe1e1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113072025-04-30T06:43:25.644ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
113082025-04-30T06:43:25.644ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 837c22cf-bcb8-420d-8b3a-390c0efda3dc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113092025-04-30T06:43:25.645ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 837c22cf-bcb8-420d-8b3a-390c0efda3dc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113102025-04-30T06:43:25.647ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 837c22cf-bcb8-420d-8b3a-390c0efda3dc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113112025-04-30T06:43:25.654ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 10856 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 837c22cf-bcb8-420d-8b3a-390c0efda3dc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113122025-04-30T06:43:25.655ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "837c22cf-bcb8-420d-8b3a-390c0efda3dc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113132025-04-30T06:43:25.655ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113142025-04-30T06:43:25.657ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 49d4fbc9-52d8-4147-91c5-472cc71d1a36 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113152025-04-30T06:43:25.658ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 49d4fbc9-52d8-4147-91c5-472cc71d1a36 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113162025-04-30T06:43:25.658ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 49d4fbc9-52d8-4147-91c5-472cc71d1a36 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113172025-04-30T06:43:25.660ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 49d4fbc9-52d8-4147-91c5-472cc71d1a36 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113182025-04-30T06:43:25.662ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 4700 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 49d4fbc9-52d8-4147-91c5-472cc71d1a36 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
113192025-04-30T06:43:25.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b56de4ea-c0a0-4bae-b4c4-9867ca906119", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113202025-04-30T06:43:25.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
113212025-04-30T06:43:25.664ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d4fbc9-52d8-4147-91c5-472cc71d1a36", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113222025-04-30T06:43:25.664ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113232025-04-30T06:43:25.664ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
113242025-04-30T06:43:25.664ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
113252025-04-30T06:43:25.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50539485-913a-4f5f-ac7f-4105bfdbcd83", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113262025-04-30T06:43:25.669ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
113272025-04-30T06:43:25.674ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30d7c6ae-19ef-4ae8-a925-c56430a2339d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113282025-04-30T06:43:25.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
113292025-04-30T06:43:25.679ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
113302025-04-30T06:43:25.683ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85926637-6ccd-455a-9f68-1c583913ad17", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113312025-04-30T06:43:25.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
113322025-04-30T06:43:25.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef0e0a7-fb99-4cf8-89d9-77240c262db2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113332025-04-30T06:43:25.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
113342025-04-30T06:43:25.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a77e2fa1-9866-4f95-94d8-4ad189bdabbb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113352025-04-30T06:43:25.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
113362025-04-30T06:43:25.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "635c1c90-4811-4c46-a5ea-05d48ee106ab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113372025-04-30T06:43:25.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
113382025-04-30T06:43:25.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed57f196-fe9b-40ea-b3cc-3186205ebe78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113392025-04-30T06:43:25.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
113402025-04-30T06:43:25.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f894ba04-6c01-4817-a31b-c2a25ea649a0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113412025-04-30T06:43:25.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
113422025-04-30T06:43:25.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95ac0fe1-ad51-48f3-80d6-211f6d2342c7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113432025-04-30T06:43:25.693ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
113442025-04-30T06:43:25.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d428c64-f111-4924-9dd9-7413d8ccbf78", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113452025-04-30T06:43:25.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
113462025-04-30T06:43:25.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b303c473-f7ef-4947-83d8-ed57788b4b67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113472025-04-30T06:43:25.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
113482025-04-30T06:43:25.704ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113492025-04-30T06:43:25.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113502025-04-30T06:43:25.720ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113512025-04-30T06:43:25.721ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113522025-04-30T06:43:25.721ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113532025-04-30T06:43:25.721ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113542025-04-30T06:43:25.730ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113552025-04-30T06:43:25.730ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113562025-04-30T06:43:25.730ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113572025-04-30T06:43:25.730ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
113582025-04-30T06:43:25.731ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113592025-04-30T06:43:25.734ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b3cde27-b200-4d52-9723-c6f3a1e18777", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113602025-04-30T06:43:25.734ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
113612025-04-30T06:43:25.766ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113622025-04-30T06:43:25.773ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113632025-04-30T06:43:25.773ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113642025-04-30T06:43:25.773ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
113652025-04-30T06:43:25.774ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113662025-04-30T06:43:25.806ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113672025-04-30T06:43:25.806ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
113682025-04-30T06:43:25.806ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113692025-04-30T06:43:25.807ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be2a43a5-cf89-4dc0-80b4-149e2c35a5ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113702025-04-30T06:43:25.808ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
113712025-04-30T06:43:25.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25f3936c-1684-4cb7-92e6-be3abfc6eb53", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113722025-04-30T06:43:25.810ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
113732025-04-30T06:43:25.818ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "678b10b8-0190-4188-89be-68864c8c5914", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113742025-04-30T06:43:25.818ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
113752025-04-30T06:43:25.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db90d294-e7be-4380-adcb-f74f1abf6434", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113762025-04-30T06:43:25.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
113772025-04-30T06:43:25.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab5d513d-27e9-4169-988d-6df90e36f186", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113782025-04-30T06:43:25.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
113792025-04-30T06:43:25.863ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f063c8d5-1f23-4589-a679-6c7bb71d099b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113802025-04-30T06:43:25.863ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
113812025-04-30T06:43:25.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb80ff68-d209-4ea1-9831-7a83f34862cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113822025-04-30T06:43:25.865ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
113832025-04-30T06:43:25.867ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df62ba3-a3cb-47be-982c-0b824a98c039", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113842025-04-30T06:43:25.867ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
113852025-04-30T06:43:25.869ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cbe69f7-6f30-42c4-b777-bbe61d56cec3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113862025-04-30T06:43:25.869ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
113872025-04-30T06:43:25.873ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dabcf3b-aa11-4550-85da-1fa865bf7ba1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113882025-04-30T06:43:25.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
113892025-04-30T06:43:25.877ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
113902025-04-30T06:43:25.880ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24acb34a-e71e-40e6-a3c9-62f1710b6fb1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113912025-04-30T06:43:25.880ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
113922025-04-30T06:43:25.882ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "182fcf57-bc07-4df0-93e3-33cd570e3058", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113932025-04-30T06:43:25.882ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
113942025-04-30T06:43:25.885ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6118a75d-75af-49f4-878d-9313620ec875", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113952025-04-30T06:43:25.886ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
113962025-04-30T06:43:25.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77d6c622-7d4f-4460-8009-8a3b96285aa8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
113972025-04-30T06:43:25.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
113982025-04-30T06:43:25.891ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
113992025-04-30T06:43:25.891ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
114002025-04-30T06:43:25.893ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
114012025-04-30T06:43:25.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd8b7a51-85c7-46ff-bb18-9975c55f23ee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114022025-04-30T06:43:25.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
114032025-04-30T06:43:25.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9eb77af-4a15-4250-ad2a-5f28edcc04f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114042025-04-30T06:43:25.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
114052025-04-30T06:43:25.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79b76f44-b9eb-49c3-aff9-555f808da77e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114062025-04-30T06:43:25.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
114072025-04-30T06:43:25.896ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb1860ce-75cc-4983-9183-2135f154919e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114082025-04-30T06:43:25.896ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
114092025-04-30T06:43:25.898ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29ec2eb-a3d0-4d64-a006-8f3ea831d926", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114102025-04-30T06:43:25.898ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
114112025-04-30T06:43:25.900ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c15b1748-d698-433a-98f1-bd1555608d9f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114122025-04-30T06:43:25.900ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
114132025-04-30T06:43:25.902ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7ef75a-8170-4eeb-a752-1bd393740b1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114142025-04-30T06:43:25.902ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
114152025-04-30T06:43:25.913ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f2a7eaf-328b-4f6c-9cbc-7f5fd4b8eac7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114162025-04-30T06:43:25.913ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
114172025-04-30T06:43:25.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f94c688-eeee-49ae-ba27-ea11a49acbea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114182025-04-30T06:43:25.916ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
114192025-04-30T06:43:25.919ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb8a6d19-d96e-484a-81ee-fede88c84d74", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114202025-04-30T06:43:25.919ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
114212025-04-30T06:43:25.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5059bd7-c0d4-4eb4-851a-0683ab5e2f9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114222025-04-30T06:43:25.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
114232025-04-30T06:43:25.930ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114242025-04-30T06:43:25.930ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114252025-04-30T06:43:25.930ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114262025-04-30T06:43:25.930ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
114272025-04-30T06:43:25.931ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114282025-04-30T06:43:25.938ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114292025-04-30T06:43:25.943ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114302025-04-30T06:43:25.943ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114312025-04-30T06:43:25.943ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
114322025-04-30T06:43:25.944ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114332025-04-30T06:43:25.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c4b0c0d-cbe7-4ebc-a91c-9f8115ff2be2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114342025-04-30T06:43:25.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
114352025-04-30T06:43:25.945ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f0925d-e0fc-4957-8ae5-75931816df8a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114362025-04-30T06:43:25.946ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
114372025-04-30T06:43:25.946ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114382025-04-30T06:43:25.946ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
114392025-04-30T06:43:25.947ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114402025-04-30T06:43:25.948ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be78ba39-b588-47d0-8234-cf00802a7e04", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114412025-04-30T06:43:25.949ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
114422025-04-30T06:43:25.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4593a23f-bd86-4ae6-aa36-330f64ada479", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114432025-04-30T06:43:25.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
114442025-04-30T06:43:25.953ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5662efa4-ba20-493a-babf-75b6a90f58c1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114452025-04-30T06:43:25.953ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114462025-04-30T06:43:25.953ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114472025-04-30T06:43:25.953ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114482025-04-30T06:43:25.953ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114492025-04-30T06:43:25.953ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114502025-04-30T06:43:25.960ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114512025-04-30T06:43:25.965ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114522025-04-30T06:43:25.965ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114532025-04-30T06:43:25.965ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114542025-04-30T06:43:25.965ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114552025-04-30T06:43:25.965ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114562025-04-30T06:43:25.966ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114572025-04-30T06:43:25.969ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114582025-04-30T06:43:25.974ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114592025-04-30T06:43:25.974ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114602025-04-30T06:43:25.974ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114612025-04-30T06:43:25.974ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114622025-04-30T06:43:25.981ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114632025-04-30T06:43:25.981ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114642025-04-30T06:43:25.981ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
114652025-04-30T06:43:25.983ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
114662025-04-30T06:43:26.057ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
114672025-04-30T06:43:26.057ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
114682025-04-30T06:43:26.058ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:62430 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
114692025-04-30T06:43:26.059ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
114702025-04-30T06:43:26.059ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
114712025-04-30T06:43:26.060ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:62430 req_id = 6816053c-6f31-450e-a0f0-29445ca3619e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114722025-04-30T06:43:26.061ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1401 local_addr = [::1]:45714 method = GET remote_addr = [::1]:62430 req_id = 6816053c-6f31-450e-a0f0-29445ca3619e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114732025-04-30T06:43:26.066ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6816053c-6f31-450e-a0f0-29445ca3619e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
114742025-04-30T06:43:26.070ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114752025-04-30T06:43:26.071ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114762025-04-30T06:43:26.071ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
114772025-04-30T06:43:26.079ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114782025-04-30T06:43:26.079ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114792025-04-30T06:43:26.079ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114802025-04-30T06:43:26.079ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114812025-04-30T06:43:26.081ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114822025-04-30T06:43:26.084ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
114832025-04-30T06:43:26.084ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114842025-04-30T06:43:26.084ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
114852025-04-30T06:43:26.086ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114862025-04-30T06:43:26.091ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114872025-04-30T06:43:26.091ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114882025-04-30T06:43:26.092ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114892025-04-30T06:43:26.092ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114902025-04-30T06:43:26.096ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
114912025-04-30T06:43:26.165ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114922025-04-30T06:43:26.165ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114932025-04-30T06:43:26.165ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114942025-04-30T06:43:26.165ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
114952025-04-30T06:43:26.167ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114962025-04-30T06:43:26.203ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114972025-04-30T06:43:26.208ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114982025-04-30T06:43:26.208ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114992025-04-30T06:43:26.209ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
115002025-04-30T06:43:26.209ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115012025-04-30T06:43:26.235ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115022025-04-30T06:43:26.235ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
115032025-04-30T06:43:26.236ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115042025-04-30T06:43:26.236ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
115052025-04-30T06:43:26.237ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
115062025-04-30T06:43:26.237ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 798.430369ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
115072025-04-30T06:43:26.237ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
115082025-04-30T06:43:26.237ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115092025-04-30T06:43:26.237ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115102025-04-30T06:43:26.237ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115112025-04-30T06:43:26.237ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115122025-04-30T06:43:26.237ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115132025-04-30T06:43:26.245ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115142025-04-30T06:43:26.249ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115152025-04-30T06:43:26.249ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115162025-04-30T06:43:26.249ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115172025-04-30T06:43:26.249ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115182025-04-30T06:43:26.255ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
115192025-04-30T06:43:26.255ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
115202025-04-30T06:43:26.257ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8b 14 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62392 req_id = ddf21307-6f7f-4047-8237-c0c83c6b4f2a
115212025-04-30T06:43:26.257ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35604,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62392 req_id = ddf21307-6f7f-4047-8237-c0c83c6b4f2a
115222025-04-30T06:43:26.258ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
115232025-04-30T06:43:26.258ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115242025-04-30T06:43:26.258ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62392 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ddf21307-6f7f-4047-8237-c0c83c6b4f2a
115252025-04-30T06:43:26.259ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 58171211 } })
115262025-04-30T06:43:26.259ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
115272025-04-30T06:43:26.268ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1f cd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49315 req_id = 880021a7-6425-4efc-a774-658c402ddb59
115282025-04-30T06:43:26.268ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8141,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49315 req_id = 880021a7-6425-4efc-a774-658c402ddb59
115292025-04-30T06:43:26.268ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115302025-04-30T06:43:26.268ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49315 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 880021a7-6425-4efc-a774-658c402ddb59
115312025-04-30T06:43:26.339ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115322025-04-30T06:43:26.346ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 85 30 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .0..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60158 req_id = 505a9140-c7f9-430c-896d-85cfcadb24bc
115332025-04-30T06:43:26.346ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34096,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60158 req_id = 505a9140-c7f9-430c-896d-85cfcadb24bc
115342025-04-30T06:43:26.347ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
115352025-04-30T06:43:26.347ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34096, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60158 req_id = 505a9140-c7f9-430c-896d-85cfcadb24bc
115362025-04-30T06:43:26.347ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
115372025-04-30T06:43:26.347ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
115382025-04-30T06:43:26.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
115392025-04-30T06:43:26.368ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115402025-04-30T06:43:26.378ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2e 70 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .p..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64903 req_id = 5c0d3054-a178-4466-857b-5b70c5ebe65f
115412025-04-30T06:43:26.378ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11888,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64903 req_id = 5c0d3054-a178-4466-857b-5b70c5ebe65f
115422025-04-30T06:43:26.378ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
115432025-04-30T06:43:26.378ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115442025-04-30T06:43:26.378ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64903 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5c0d3054-a178-4466-857b-5b70c5ebe65f
115452025-04-30T06:43:26.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc8199a8-38af-446a-8400-88b678655018", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
115462025-04-30T06:43:26.379ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
115472025-04-30T06:43:26.384ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 183542315 } })
115482025-04-30T06:43:26.384ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
115492025-04-30T06:43:26.388ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
115502025-04-30T06:43:26.394ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a0 63 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .c..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50180 req_id = 76648f79-3052-48a8-ade6-a3c79991c446
115512025-04-30T06:43:26.395ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41059,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50180 req_id = 76648f79-3052-48a8-ade6-a3c79991c446
115522025-04-30T06:43:26.395ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115532025-04-30T06:43:26.395ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50180 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 76648f79-3052-48a8-ade6-a3c79991c446
115542025-04-30T06:43:26.398ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
115552025-04-30T06:43:26.488ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
115562025-04-30T06:43:26.488ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
115572025-04-30T06:43:26.490ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
115582025-04-30T06:43:26.490ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
115592025-04-30T06:43:26.491ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
115602025-04-30T06:43:26.491ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115612025-04-30T06:43:26.492ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115622025-04-30T06:43:26.492ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115632025-04-30T06:43:26.493ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
115642025-04-30T06:43:26.493ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115652025-04-30T06:43:26.494ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115662025-04-30T06:43:26.495ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afda1250-f121-4d92-8650-6aee89508380", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
115672025-04-30T06:43:26.496ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
115682025-04-30T06:43:26.496ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 42 17 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 B............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61434 req_id = 23f4e645-71fd-41a5-bc5f-86e0bb3f3b3e
115692025-04-30T06:43:26.497ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16919,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61434 req_id = 23f4e645-71fd-41a5-bc5f-86e0bb3f3b3e
115702025-04-30T06:43:26.497ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
115712025-04-30T06:43:26.497ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115722025-04-30T06:43:26.511ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61434 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 23f4e645-71fd-41a5-bc5f-86e0bb3f3b3e
115732025-04-30T06:43:26.514ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115742025-04-30T06:43:26.523ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115752025-04-30T06:43:26.523ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115762025-04-30T06:43:26.523ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
115772025-04-30T06:43:26.524ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115782025-04-30T06:43:26.525ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 324072442 } })
115792025-04-30T06:43:26.525ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
115802025-04-30T06:43:26.526ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e 4c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .L..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60305 req_id = 35aabd31-259d-4862-88ab-9b8d188fdcff
115812025-04-30T06:43:26.526ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40524,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60305 req_id = 35aabd31-259d-4862-88ab-9b8d188fdcff
115822025-04-30T06:43:26.527ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115832025-04-30T06:43:26.527ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60305 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 35aabd31-259d-4862-88ab-9b8d188fdcff
115842025-04-30T06:43:26.527ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
115852025-04-30T06:43:26.606ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
115862025-04-30T06:43:26.607ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
115872025-04-30T06:43:26.608ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
115882025-04-30T06:43:26.608ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
115892025-04-30T06:43:26.609ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
115902025-04-30T06:43:26.610ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115912025-04-30T06:43:26.610ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
115922025-04-30T06:43:26.611ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115932025-04-30T06:43:26.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "382d0fda-3a7d-46c2-ab47-a4b4fd6f0196", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:25 GMT"} })
115942025-04-30T06:43:26.615ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
115952025-04-30T06:43:26.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0420378-e8a9-4047-b356-ad446c54e11d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
115962025-04-30T06:43:26.628ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
115972025-04-30T06:43:26.632ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34ad3bad-dddb-48d6-a7ed-49ed946478be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
115982025-04-30T06:43:26.632ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
115992025-04-30T06:43:26.639ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d0311f-839a-4478-9c03-4a91d4d355a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116002025-04-30T06:43:26.640ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
116012025-04-30T06:43:26.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5817b4bd-3daa-460a-9d82-210999164247", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116022025-04-30T06:43:26.649ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
116032025-04-30T06:43:26.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55a4a99-86d7-4caf-a866-4236c6f0ab14", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116042025-04-30T06:43:26.654ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
116052025-04-30T06:43:26.655ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
116062025-04-30T06:43:26.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8452a849-4958-4989-abbd-b6395a042247", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116072025-04-30T06:43:26.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
116082025-04-30T06:43:26.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed0b79c4-37f1-4646-9d21-96d602e676de", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116092025-04-30T06:43:26.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
116102025-04-30T06:43:26.660ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3e65442-442f-491b-ad04-48ff1c850596", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116112025-04-30T06:43:26.660ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
116122025-04-30T06:43:26.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a47acaf7-d119-4811-a28b-84f3f5d0e350", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116132025-04-30T06:43:26.661ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
116142025-04-30T06:43:26.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "376c361c-6072-401e-b030-3cb480293906", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116152025-04-30T06:43:26.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
116162025-04-30T06:43:26.666ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcfccf02-aa80-4e4f-9c76-42afb4a29357", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116172025-04-30T06:43:26.666ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
116182025-04-30T06:43:26.668ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22be155b-3091-4907-92d5-439973b089a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116192025-04-30T06:43:26.668ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
116202025-04-30T06:43:26.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "919a21ef-f5fa-4b8e-b599-a8bcef519341", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116212025-04-30T06:43:26.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
116222025-04-30T06:43:26.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67056145-a022-459d-8108-2e3cc4d35e0f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116232025-04-30T06:43:26.694ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
116242025-04-30T06:43:26.696ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54edd15-44b1-4d9b-8c6e-95ac91937e0d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116252025-04-30T06:43:26.696ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
116262025-04-30T06:43:26.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b77b949a-2fcc-413f-ba3e-bc1e8374e6de", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116272025-04-30T06:43:26.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
116282025-04-30T06:43:26.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75e1f387-6b15-43c9-8913-12f7b6d7887c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116292025-04-30T06:43:26.700ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
116302025-04-30T06:43:26.701ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bcff1d3-d149-4cc3-b428-6db339167fdb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116312025-04-30T06:43:26.701ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
116322025-04-30T06:43:26.702ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fab9c1de-78f1-4033-8df3-ed42041f70cf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116332025-04-30T06:43:26.703ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
116342025-04-30T06:43:26.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116352025-04-30T06:43:26.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116362025-04-30T06:43:26.705ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdeb31dd-67ea-404d-90fc-c662522ca8d5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116372025-04-30T06:43:26.705ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
116382025-04-30T06:43:26.706ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116392025-04-30T06:43:26.707ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116402025-04-30T06:43:26.707ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116412025-04-30T06:43:26.707ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
116422025-04-30T06:43:26.708ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116432025-04-30T06:43:26.709ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116442025-04-30T06:43:26.718ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116452025-04-30T06:43:26.718ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116462025-04-30T06:43:26.718ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116472025-04-30T06:43:26.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29634d4d-a5fc-40cc-93d7-48113eed7858", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116482025-04-30T06:43:26.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
116492025-04-30T06:43:26.722ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116502025-04-30T06:43:26.728ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116512025-04-30T06:43:26.729ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116522025-04-30T06:43:26.729ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
116532025-04-30T06:43:26.729ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116542025-04-30T06:43:26.732ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116552025-04-30T06:43:26.732ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
116562025-04-30T06:43:26.733ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116572025-04-30T06:43:26.738ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "316126a2-c4f4-499c-9ff6-122aee2ba3c1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116582025-04-30T06:43:26.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
116592025-04-30T06:43:26.741ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f3720d3-63cc-4dbd-99b7-77e29553f1b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116602025-04-30T06:43:26.741ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
116612025-04-30T06:43:26.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9cb7906-63aa-4161-ac23-5b12839ac6d7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116622025-04-30T06:43:26.748ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
116632025-04-30T06:43:26.750ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c48e8ac-5b79-4ef8-ae11-90188783fa6f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116642025-04-30T06:43:26.750ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
116652025-04-30T06:43:26.753ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53ccfcd2-aeea-4059-9a81-ac47bf0fd6bd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116662025-04-30T06:43:26.754ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
116672025-04-30T06:43:26.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9acc2cc-2308-4b64-9d06-b085617b2de1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116682025-04-30T06:43:26.756ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
116692025-04-30T06:43:26.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c6a64f-1ef2-413d-8d0a-3bf6669ae679", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116702025-04-30T06:43:26.759ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
116712025-04-30T06:43:26.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa3d6492-9504-463a-a111-9ac488dae898", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116722025-04-30T06:43:26.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
116732025-04-30T06:43:26.766ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2400e106-08b8-4a80-8c31-11f18b74786a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116742025-04-30T06:43:26.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
116752025-04-30T06:43:26.767ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
116762025-04-30T06:43:26.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e27a969f-602c-4214-86a9-fb0aeb136c8f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116772025-04-30T06:43:26.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
116782025-04-30T06:43:26.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20fa3514-3232-4488-9e38-83b91b71b460", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116792025-04-30T06:43:26.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
116802025-04-30T06:43:26.772ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4658c5f0-5543-4783-a164-75fc85366ae8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116812025-04-30T06:43:26.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
116822025-04-30T06:43:26.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "020be5ff-3124-417f-a5d2-34773615e6bd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116832025-04-30T06:43:26.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
116842025-04-30T06:43:26.777ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59f225cf-827f-4551-843a-c7ab01cf0a67", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116852025-04-30T06:43:26.778ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
116862025-04-30T06:43:26.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64eee5c3-d698-4fed-b84f-810db81bf3e0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116872025-04-30T06:43:26.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
116882025-04-30T06:43:26.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16df8f46-3310-4e51-abd7-151afadba0d5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116892025-04-30T06:43:26.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
116902025-04-30T06:43:26.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c6c36f-e7d5-4bec-9115-d68c46831093", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116912025-04-30T06:43:26.788ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
116922025-04-30T06:43:26.791ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3817263-2770-4403-975f-e71e0c1bde2a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116932025-04-30T06:43:26.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
116942025-04-30T06:43:26.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5df5a5c5-2d1e-4754-aca4-f66926cc83fe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116952025-04-30T06:43:26.797ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
116962025-04-30T06:43:26.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d0687bf-8a22-4efc-a9cb-52090e3b2e37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116972025-04-30T06:43:26.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
116982025-04-30T06:43:26.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c20c10-01c8-4cd4-a26c-3672064fbe39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
116992025-04-30T06:43:26.803ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117002025-04-30T06:43:26.803ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117012025-04-30T06:43:26.803ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117022025-04-30T06:43:26.803ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117032025-04-30T06:43:26.805ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117042025-04-30T06:43:26.814ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117052025-04-30T06:43:26.820ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117062025-04-30T06:43:26.820ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117072025-04-30T06:43:26.820ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117082025-04-30T06:43:26.820ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117092025-04-30T06:43:26.820ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117102025-04-30T06:43:26.821ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117112025-04-30T06:43:26.823ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117122025-04-30T06:43:26.823ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117132025-04-30T06:43:26.823ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117142025-04-30T06:43:26.823ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
117152025-04-30T06:43:26.823ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117162025-04-30T06:43:26.825ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117172025-04-30T06:43:26.829ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117182025-04-30T06:43:26.829ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117192025-04-30T06:43:26.829ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117202025-04-30T06:43:26.830ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117212025-04-30T06:43:26.831ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117222025-04-30T06:43:26.841ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117232025-04-30T06:43:26.841ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117242025-04-30T06:43:26.841ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
117252025-04-30T06:43:26.842ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117262025-04-30T06:43:26.845ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117272025-04-30T06:43:26.845ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
117282025-04-30T06:43:26.845ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117292025-04-30T06:43:26.915ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
117302025-04-30T06:43:26.915ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
117312025-04-30T06:43:26.916ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:65039 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117322025-04-30T06:43:26.917ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:65039 req_id = 0d244f6d-09b0-43de-bca6-c18f861c570a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117332025-04-30T06:43:26.917ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 470 local_addr = [::1]:45714 method = GET remote_addr = [::1]:65039 req_id = 0d244f6d-09b0-43de-bca6-c18f861c570a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117342025-04-30T06:43:26.918ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d244f6d-09b0-43de-bca6-c18f861c570a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
117352025-04-30T06:43:26.921ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117362025-04-30T06:43:26.921ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117372025-04-30T06:43:26.921ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
117382025-04-30T06:43:26.923ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117392025-04-30T06:43:26.923ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117402025-04-30T06:43:26.923ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117412025-04-30T06:43:26.923ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117422025-04-30T06:43:26.924ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117432025-04-30T06:43:26.927ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117442025-04-30T06:43:26.933ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117452025-04-30T06:43:26.933ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117462025-04-30T06:43:26.933ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117472025-04-30T06:43:26.934ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117482025-04-30T06:43:26.976ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117492025-04-30T06:43:26.976ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117502025-04-30T06:43:26.976ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
117512025-04-30T06:43:26.978ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
117522025-04-30T06:43:26.982ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
117532025-04-30T06:43:26.982ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
117542025-04-30T06:43:26.982ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
117552025-04-30T06:43:26.987ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
117562025-04-30T06:43:26.987ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117572025-04-30T06:43:26.990ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117582025-04-30T06:43:26.990ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117592025-04-30T06:43:26.991ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 753.658526ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117602025-04-30T06:43:26.991ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
117612025-04-30T06:43:26.991ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117622025-04-30T06:43:26.991ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117632025-04-30T06:43:26.991ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117642025-04-30T06:43:26.991ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117652025-04-30T06:43:26.992ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117662025-04-30T06:43:26.994ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117672025-04-30T06:43:27.001ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117682025-04-30T06:43:27.001ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117692025-04-30T06:43:27.001ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117702025-04-30T06:43:27.002ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117712025-04-30T06:43:27.005ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
117722025-04-30T06:43:27.005ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117732025-04-30T06:43:27.008ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 75 e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 u............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63085 req_id = e3e5b5ad-3899-4fac-ad4c-d932871469cb
117742025-04-30T06:43:27.008ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30176,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63085 req_id = e3e5b5ad-3899-4fac-ad4c-d932871469cb
117752025-04-30T06:43:27.008ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
117762025-04-30T06:43:27.009ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117772025-04-30T06:43:27.009ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63085 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e3e5b5ad-3899-4fac-ad4c-d932871469cb
117782025-04-30T06:43:27.011ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4271, tv_nsec: 808969558 } })
117792025-04-30T06:43:27.011ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
117802025-04-30T06:43:27.011ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 1c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61343 req_id = 6ef6457b-b541-4cde-9296-d6398d563f6e
117812025-04-30T06:43:27.011ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49948,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61343 req_id = 6ef6457b-b541-4cde-9296-d6398d563f6e
117822025-04-30T06:43:27.011ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117832025-04-30T06:43:27.011ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61343 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6ef6457b-b541-4cde-9296-d6398d563f6e
117842025-04-30T06:43:27.120ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
117852025-04-30T06:43:27.121ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117862025-04-30T06:43:27.121ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117872025-04-30T06:43:27.121ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117882025-04-30T06:43:27.121ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
117892025-04-30T06:43:27.122ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117902025-04-30T06:43:27.126ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
117912025-04-30T06:43:27.126ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6a 8c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c j............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58407 req_id = 375c20b4-6e58-4ad6-8683-1480c3f3dd4d
117922025-04-30T06:43:27.126ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27276,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58407 req_id = 375c20b4-6e58-4ad6-8683-1480c3f3dd4d
117932025-04-30T06:43:27.126ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
117942025-04-30T06:43:27.126ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27276, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58407 req_id = 375c20b4-6e58-4ad6-8683-1480c3f3dd4d
117952025-04-30T06:43:27.126ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
117962025-04-30T06:43:27.126ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
117972025-04-30T06:43:27.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
117982025-04-30T06:43:27.128ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117992025-04-30T06:43:27.135ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118002025-04-30T06:43:27.135ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118012025-04-30T06:43:27.135ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
118022025-04-30T06:43:27.137ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118032025-04-30T06:43:27.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ae3c702-a4ea-482b-8b1d-c36cbf7ef2c1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118042025-04-30T06:43:27.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
118052025-04-30T06:43:27.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ffe6200-211a-47e5-8a5b-edc4a13ac296", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118062025-04-30T06:43:27.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
118072025-04-30T06:43:27.142ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118082025-04-30T06:43:27.143ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
118092025-04-30T06:43:27.143ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118102025-04-30T06:43:27.146ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "368d936f-827d-4fc2-bafd-7fc949c74907", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118112025-04-30T06:43:27.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
118122025-04-30T06:43:27.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcc33011-74e4-474c-86d5-371b68c9db4a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118132025-04-30T06:43:27.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
118142025-04-30T06:43:27.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72613f79-d4cb-45a9-aa21-d5949a1d9b34", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118152025-04-30T06:43:27.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
118162025-04-30T06:43:27.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8261132f-f500-44a6-b78b-df2eb7502fe7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118172025-04-30T06:43:27.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
118182025-04-30T06:43:27.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f4a791-bcdc-4ad5-92fe-cb2a1360bfb1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118192025-04-30T06:43:27.178ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
118202025-04-30T06:43:27.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3698e8-5ac2-441b-9ba1-0d53650d7ef6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118212025-04-30T06:43:27.182ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
118222025-04-30T06:43:27.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4eaae0-bfde-45a9-a568-5bb89849c2f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118232025-04-30T06:43:27.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
118242025-04-30T06:43:27.189ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
118252025-04-30T06:43:27.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74d70f04-c002-49a8-9068-f990b1912b35", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118262025-04-30T06:43:27.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
118272025-04-30T06:43:27.193ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9533b06c-d0cb-4d18-984d-c2d79bc1b625", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118282025-04-30T06:43:27.193ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
118292025-04-30T06:43:27.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b422cd66-8e5e-4696-a55e-0aeb74c23d7c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118302025-04-30T06:43:27.194ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
118312025-04-30T06:43:27.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6903a8a-6f55-4ac5-9764-10e9de07e59c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118322025-04-30T06:43:27.197ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
118332025-04-30T06:43:27.200ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89184115-8095-4540-82b0-03141fa0389f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118342025-04-30T06:43:27.201ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
118352025-04-30T06:43:27.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec45593-f9ae-4ccc-981f-c85d3abf84df", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118362025-04-30T06:43:27.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
118372025-04-30T06:43:27.208ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c64b9c06-635e-4f1f-bdf3-dfc059a928da", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118382025-04-30T06:43:27.208ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
118392025-04-30T06:43:27.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06329bc3-a14a-4f52-98b8-d0b862784ba9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118402025-04-30T06:43:27.210ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
118412025-04-30T06:43:27.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7702edd5-ceb6-4258-a50e-93edb5087199", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118422025-04-30T06:43:27.211ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
118432025-04-30T06:43:27.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb85cd07-b93e-44e8-81ea-ad2305ab6ab9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118442025-04-30T06:43:27.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
118452025-04-30T06:43:27.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91292ffe-32e1-4513-bc28-e0322eae65fd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118462025-04-30T06:43:27.214ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
118472025-04-30T06:43:27.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea8c678-a397-4768-9809-b42fd1e67cdc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118482025-04-30T06:43:27.215ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
118492025-04-30T06:43:27.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29ce85e9-0c95-4102-80dc-e49eda38132b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118502025-04-30T06:43:27.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
118512025-04-30T06:43:27.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb41e8da-ec48-4fc8-85d8-6a28218ccc74", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118522025-04-30T06:43:27.217ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
118532025-04-30T06:43:27.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa02648-49c8-49bb-bf07-735e5fca72cd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118542025-04-30T06:43:27.219ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
118552025-04-30T06:43:27.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d7d5fa-1896-4670-9fde-c57f96838a53", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118562025-04-30T06:43:27.223ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
118572025-04-30T06:43:27.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4ae647b-e121-4a1e-b6e1-3becdb3c8bf5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118582025-04-30T06:43:27.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
118592025-04-30T06:43:27.229ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e06238-1b79-4748-b64e-3b824a506ee4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118602025-04-30T06:43:27.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
118612025-04-30T06:43:27.239ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118622025-04-30T06:43:27.239ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118632025-04-30T06:43:27.240ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118642025-04-30T06:43:27.240ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
118652025-04-30T06:43:27.243ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118662025-04-30T06:43:27.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68137e4e-a899-4fd3-baf6-2672f3ae3935", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118672025-04-30T06:43:27.247ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
118682025-04-30T06:43:27.248ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118692025-04-30T06:43:27.257ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118702025-04-30T06:43:27.257ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118712025-04-30T06:43:27.257ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
118722025-04-30T06:43:27.258ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118732025-04-30T06:43:27.274ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118742025-04-30T06:43:27.274ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
118752025-04-30T06:43:27.275ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118762025-04-30T06:43:27.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c72da78-0590-44d8-8ab8-4f12810bafe0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118772025-04-30T06:43:27.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
118782025-04-30T06:43:27.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2703317-76a5-452b-ab3b-2f88204d51e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118792025-04-30T06:43:27.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
118802025-04-30T06:43:27.358ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
118812025-04-30T06:43:27.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3db06dd4-54cb-4483-9947-34e79b87ba49", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118822025-04-30T06:43:27.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
118832025-04-30T06:43:27.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d812cadc-8ead-46a0-b605-c94a699b50ab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118842025-04-30T06:43:27.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
118852025-04-30T06:43:27.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8dae366-1aea-4cc9-802b-8e59d514b93a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118862025-04-30T06:43:27.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
118872025-04-30T06:43:27.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b97ca4a-a6cb-43a4-bf0a-f38441a6c518", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118882025-04-30T06:43:27.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
118892025-04-30T06:43:27.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "770eb54b-8d8d-45aa-b93a-26e37163af13", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118902025-04-30T06:43:27.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
118912025-04-30T06:43:27.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e138790-29aa-450d-9f3d-54959cca4e01", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118922025-04-30T06:43:27.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
118932025-04-30T06:43:27.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "388fabe8-9f20-4b72-bfaf-03225c8c1706", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118942025-04-30T06:43:27.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
118952025-04-30T06:43:27.379ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc150d65-af05-4ad2-a15e-9b718c9fcba6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118962025-04-30T06:43:27.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
118972025-04-30T06:43:27.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d102c1f-6b75-40c6-baa0-e540a50ba90c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
118982025-04-30T06:43:27.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
118992025-04-30T06:43:27.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d9a3c1d-4391-4954-851a-d295ae3bfac2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119002025-04-30T06:43:27.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
119012025-04-30T06:43:27.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29d66f4c-7585-452d-a7cf-245ed0d6e51a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119022025-04-30T06:43:27.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
119032025-04-30T06:43:27.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c70af521-d827-48d8-90e5-85a75c876ff8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119042025-04-30T06:43:27.391ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
119052025-04-30T06:43:27.392ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3138408-0b50-407e-a904-c492e14b5b4f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119062025-04-30T06:43:27.393ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
119072025-04-30T06:43:27.394ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "761476fa-3c0d-4eb3-9676-4b11ae3b5aaf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119082025-04-30T06:43:27.394ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
119092025-04-30T06:43:27.395ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e85b3a-2b2a-415f-89df-7298b034f4db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119102025-04-30T06:43:27.395ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119112025-04-30T06:43:27.395ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119122025-04-30T06:43:27.395ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119132025-04-30T06:43:27.395ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119142025-04-30T06:43:27.396ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119152025-04-30T06:43:27.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119162025-04-30T06:43:27.415ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119172025-04-30T06:43:27.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119182025-04-30T06:43:27.415ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
119192025-04-30T06:43:27.416ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119202025-04-30T06:43:27.418ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119212025-04-30T06:43:27.426ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119222025-04-30T06:43:27.426ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119232025-04-30T06:43:27.426ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119242025-04-30T06:43:27.426ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119252025-04-30T06:43:27.426ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119262025-04-30T06:43:27.426ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119272025-04-30T06:43:27.434ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119282025-04-30T06:43:27.446ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119292025-04-30T06:43:27.446ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119302025-04-30T06:43:27.446ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
119312025-04-30T06:43:27.449ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119322025-04-30T06:43:27.450ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119332025-04-30T06:43:27.470ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119342025-04-30T06:43:27.471ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119352025-04-30T06:43:27.471ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119362025-04-30T06:43:27.472ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119372025-04-30T06:43:27.476ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119382025-04-30T06:43:27.476ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
119392025-04-30T06:43:27.480ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119402025-04-30T06:43:27.584ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
119412025-04-30T06:43:27.584ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
119422025-04-30T06:43:27.590ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:43924 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119432025-04-30T06:43:27.590ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:43924 req_id = 4f3a544d-e563-4746-bf55-9d006f06254c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119442025-04-30T06:43:27.591ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 758 local_addr = [::1]:45714 method = GET remote_addr = [::1]:43924 req_id = 4f3a544d-e563-4746-bf55-9d006f06254c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119452025-04-30T06:43:27.591ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f3a544d-e563-4746-bf55-9d006f06254c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:26 GMT"} })
119462025-04-30T06:43:27.599ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119472025-04-30T06:43:27.600ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119482025-04-30T06:43:27.600ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
119492025-04-30T06:43:27.603ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119502025-04-30T06:43:27.603ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119512025-04-30T06:43:27.603ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119522025-04-30T06:43:27.603ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119532025-04-30T06:43:27.604ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119542025-04-30T06:43:27.627ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119552025-04-30T06:43:27.640ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119562025-04-30T06:43:27.640ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119572025-04-30T06:43:27.640ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119582025-04-30T06:43:27.641ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119592025-04-30T06:43:27.642ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
119602025-04-30T06:43:27.642ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119612025-04-30T06:43:27.644ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6a5b77c4-d95a-46ed-89d6-1d0649e15d1a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119622025-04-30T06:43:27.644ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6a5b77c4-d95a-46ed-89d6-1d0649e15d1a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119632025-04-30T06:43:27.644ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6a5b77c4-d95a-46ed-89d6-1d0649e15d1a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119642025-04-30T06:43:27.645ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6a5b77c4-d95a-46ed-89d6-1d0649e15d1a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119652025-04-30T06:43:27.652ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 8866 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 6a5b77c4-d95a-46ed-89d6-1d0649e15d1a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
119662025-04-30T06:43:27.654ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5b77c4-d95a-46ed-89d6-1d0649e15d1a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
119672025-04-30T06:43:27.656ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119682025-04-30T06:43:27.656ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 74fd0995-28bd-4fa2-88e0-bc05342e416e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119692025-04-30T06:43:27.657ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 74fd0995-28bd-4fa2-88e0-bc05342e416e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119702025-04-30T06:43:27.657ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 74fd0995-28bd-4fa2-88e0-bc05342e416e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119712025-04-30T06:43:27.660ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 74fd0995-28bd-4fa2-88e0-bc05342e416e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119722025-04-30T06:43:27.664ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 7406 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 74fd0995-28bd-4fa2-88e0-bc05342e416e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
119732025-04-30T06:43:27.665ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "74fd0995-28bd-4fa2-88e0-bc05342e416e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
119742025-04-30T06:43:27.665ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119752025-04-30T06:43:27.665ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
119762025-04-30T06:43:27.665ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
119772025-04-30T06:43:27.685ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
119782025-04-30T06:43:27.705ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119792025-04-30T06:43:27.705ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119802025-04-30T06:43:27.706ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
119812025-04-30T06:43:27.706ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
119822025-04-30T06:43:27.706ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 715.671188ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
119832025-04-30T06:43:27.706ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
119842025-04-30T06:43:27.706ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119852025-04-30T06:43:27.706ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119862025-04-30T06:43:27.706ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119872025-04-30T06:43:27.706ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119882025-04-30T06:43:27.708ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119892025-04-30T06:43:27.717ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119902025-04-30T06:43:27.718ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119912025-04-30T06:43:27.718ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119922025-04-30T06:43:27.719ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119932025-04-30T06:43:27.720ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119942025-04-30T06:43:27.727ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119952025-04-30T06:43:27.727ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119962025-04-30T06:43:27.727ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119972025-04-30T06:43:27.728ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119982025-04-30T06:43:27.735ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
119992025-04-30T06:43:27.735ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120002025-04-30T06:43:27.736ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120012025-04-30T06:43:27.736ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120022025-04-30T06:43:27.736ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120032025-04-30T06:43:27.736ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
120042025-04-30T06:43:27.738ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120052025-04-30T06:43:27.738ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 59 a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Y............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61815 req_id = a67fa99f-8ddd-4c73-9294-da0f3d8c64ed
120062025-04-30T06:43:27.739ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22945,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61815 req_id = a67fa99f-8ddd-4c73-9294-da0f3d8c64ed
120072025-04-30T06:43:27.739ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
120082025-04-30T06:43:27.739ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120092025-04-30T06:43:27.739ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61815 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a67fa99f-8ddd-4c73-9294-da0f3d8c64ed
120102025-04-30T06:43:27.739ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4272, tv_nsec: 538707964 } })
120112025-04-30T06:43:27.739ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
120122025-04-30T06:43:27.740ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e7 b0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56879 req_id = bd0749f5-5fa1-4f1f-b680-867f4bd21a58
120132025-04-30T06:43:27.740ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56879 req_id = bd0749f5-5fa1-4f1f-b680-867f4bd21a58
120142025-04-30T06:43:27.740ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120152025-04-30T06:43:27.740ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56879 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bd0749f5-5fa1-4f1f-b680-867f4bd21a58
120162025-04-30T06:43:27.859ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120172025-04-30T06:43:27.866ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120182025-04-30T06:43:27.879ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120192025-04-30T06:43:27.879ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120202025-04-30T06:43:27.879ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
120212025-04-30T06:43:27.880ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120222025-04-30T06:43:27.881ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 20 f9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49942 req_id = f7b381d5-e10f-40a7-8d3e-000a8c8263b5
120232025-04-30T06:43:27.881ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8441,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49942 req_id = f7b381d5-e10f-40a7-8d3e-000a8c8263b5
120242025-04-30T06:43:27.881ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
120252025-04-30T06:43:27.881ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8441, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49942 req_id = f7b381d5-e10f-40a7-8d3e-000a8c8263b5
120262025-04-30T06:43:27.883ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
120272025-04-30T06:43:27.884ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
120282025-04-30T06:43:27.884ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
120292025-04-30T06:43:27.884ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120302025-04-30T06:43:27.884ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
120312025-04-30T06:43:27.886ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120322025-04-30T06:43:27.891ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c486a8-c4d7-437a-9d68-f98e4f0d0322", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120332025-04-30T06:43:27.891ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
120342025-04-30T06:43:27.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bee3d4a1-bbd9-4e69-b9c3-1681282ce191", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120352025-04-30T06:43:27.894ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
120362025-04-30T06:43:27.894ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
120372025-04-30T06:43:27.896ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7a 7e 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 z~..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61064 req_id = 25a7457f-2997-42ec-9ef9-68825d41466a
120382025-04-30T06:43:27.896ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31358,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61064 req_id = 25a7457f-2997-42ec-9ef9-68825d41466a
120392025-04-30T06:43:27.896ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
120402025-04-30T06:43:27.897ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120412025-04-30T06:43:27.897ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61064 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 25a7457f-2997-42ec-9ef9-68825d41466a
120422025-04-30T06:43:27.897ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4272, tv_nsec: 696844631 } })
120432025-04-30T06:43:27.897ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
120442025-04-30T06:43:27.903ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c7 5a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Z..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60892 req_id = f5c3896d-bf94-4ae0-ab4e-e7f1606d419e
120452025-04-30T06:43:27.904ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51034,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60892 req_id = f5c3896d-bf94-4ae0-ab4e-e7f1606d419e
120462025-04-30T06:43:27.904ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120472025-04-30T06:43:27.904ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60892 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f5c3896d-bf94-4ae0-ab4e-e7f1606d419e
120482025-04-30T06:43:27.905ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
120492025-04-30T06:43:27.999ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
120502025-04-30T06:43:28.000ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
120512025-04-30T06:43:28.001ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
120522025-04-30T06:43:28.001ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
120532025-04-30T06:43:28.003ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
120542025-04-30T06:43:28.004ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120552025-04-30T06:43:28.004ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120562025-04-30T06:43:28.004ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
120572025-04-30T06:43:28.006ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
120582025-04-30T06:43:28.010ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03fc4bf7-e8b3-44cf-b956-76cdf15eaa87", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120592025-04-30T06:43:28.010ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
120602025-04-30T06:43:28.012ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
120612025-04-30T06:43:28.012ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
120622025-04-30T06:43:28.020ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
120632025-04-30T06:43:28.020ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120642025-04-30T06:43:28.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cd63bb4-122d-43fc-84de-36caea0314e4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120652025-04-30T06:43:28.022ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
120662025-04-30T06:43:28.024ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24cc694d-1c87-4b0d-8582-f89b0d1adb7c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120672025-04-30T06:43:28.025ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
120682025-04-30T06:43:28.028ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06ba9a27-426b-499b-8895-5f240a786dec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120692025-04-30T06:43:28.028ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
120702025-04-30T06:43:28.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c70ce2-bd58-4b05-9343-8890fc76b082", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120712025-04-30T06:43:28.030ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
120722025-04-30T06:43:28.034ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
120732025-04-30T06:43:28.036ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f3b3bfc-eb64-4714-a805-3985e6112ea9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120742025-04-30T06:43:28.037ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
120752025-04-30T06:43:28.038ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd382eea-0eb9-4fbc-9591-23178ac99502", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120762025-04-30T06:43:28.038ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
120772025-04-30T06:43:28.039ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85fb1d8b-000c-4c57-860b-8a5f81f9d7b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120782025-04-30T06:43:28.039ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
120792025-04-30T06:43:28.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "088d0685-0b91-411b-b0f6-91f2f0c31e8b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120802025-04-30T06:43:28.041ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
120812025-04-30T06:43:28.042ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bcbec1f-3f9c-4367-bbad-8810ab40d380", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120822025-04-30T06:43:28.042ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
120832025-04-30T06:43:28.044ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b74c2627-bd34-459e-acd5-eae4d04462bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120842025-04-30T06:43:28.044ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
120852025-04-30T06:43:28.049ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49831e13-89d2-4718-a474-c214bc342970", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120862025-04-30T06:43:28.050ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
120872025-04-30T06:43:28.054ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ddd021-6b89-4a34-8d24-7548e07109af", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120882025-04-30T06:43:28.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
120892025-04-30T06:43:28.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c14372a1-8ddd-4af9-b697-767336d46662", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120902025-04-30T06:43:28.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
120912025-04-30T06:43:28.062ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd0e15ee-ed8c-459e-bcab-1d54a450cb8c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120922025-04-30T06:43:28.062ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
120932025-04-30T06:43:28.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "127febb2-f8a1-4c58-9612-3e168206e3bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120942025-04-30T06:43:28.067ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
120952025-04-30T06:43:28.072ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cef8e96-dd9e-4212-8bb7-2263012885e0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120962025-04-30T06:43:28.072ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
120972025-04-30T06:43:28.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc3b396-c375-4206-8805-7d0d961330fe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
120982025-04-30T06:43:28.073ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
120992025-04-30T06:43:28.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81eef835-bdcc-45f5-b6dc-c0644cb560ec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121002025-04-30T06:43:28.076ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
121012025-04-30T06:43:28.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bbd9395-1309-433e-8b39-d279bb4b62bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121022025-04-30T06:43:28.077ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
121032025-04-30T06:43:28.081ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46862d6-d4e2-451c-aed9-4f08cdc5006d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121042025-04-30T06:43:28.081ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
121052025-04-30T06:43:28.086ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121062025-04-30T06:43:28.086ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121072025-04-30T06:43:28.086ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121082025-04-30T06:43:28.086ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
121092025-04-30T06:43:28.088ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121102025-04-30T06:43:28.091ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "881194c5-8ae0-46bd-bb08-10e38ad00a3e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121112025-04-30T06:43:28.091ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
121122025-04-30T06:43:28.091ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121132025-04-30T06:43:28.098ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121142025-04-30T06:43:28.098ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121152025-04-30T06:43:28.098ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
121162025-04-30T06:43:28.099ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121172025-04-30T06:43:28.102ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121182025-04-30T06:43:28.102ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
121192025-04-30T06:43:28.103ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121202025-04-30T06:43:28.106ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
121212025-04-30T06:43:28.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80f55afb-b62b-4932-806a-18f0536076b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121222025-04-30T06:43:28.107ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
121232025-04-30T06:43:28.109ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76f4964b-cc0b-4bc6-a87c-2f2328351a04", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121242025-04-30T06:43:28.109ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
121252025-04-30T06:43:28.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bc1be7f-5514-4782-915d-f3fc0a2fa674", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121262025-04-30T06:43:28.112ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
121272025-04-30T06:43:28.115ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d09810-d58d-42b2-b6d5-b3f2fa1813ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121282025-04-30T06:43:28.115ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
121292025-04-30T06:43:28.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db055eb2-429e-41c9-8375-08eb17e0d7fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121302025-04-30T06:43:28.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
121312025-04-30T06:43:28.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "085c9eb7-b2f8-47d7-8205-7c418159a76c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121322025-04-30T06:43:28.121ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
121332025-04-30T06:43:28.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "952abfbf-95cd-459c-909f-f32f8764ffed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121342025-04-30T06:43:28.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
121352025-04-30T06:43:28.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c8e03b-7378-42c7-b743-7f93ca081874", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121362025-04-30T06:43:28.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
121372025-04-30T06:43:28.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37159871-9f3b-413a-8b2d-d7171b35a69a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121382025-04-30T06:43:28.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
121392025-04-30T06:43:28.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef9225de-f8f8-4251-b54d-270e779f9d7f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121402025-04-30T06:43:28.129ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
121412025-04-30T06:43:28.131ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b89af66a-d8fe-4b3e-9eca-4476e1938b7d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121422025-04-30T06:43:28.132ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
121432025-04-30T06:43:28.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1b3ea3c-ea5d-4a92-a55e-b207a45d17b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121442025-04-30T06:43:28.134ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
121452025-04-30T06:43:28.137ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d483d345-445f-417d-9642-3f5c0a76db47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121462025-04-30T06:43:28.138ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
121472025-04-30T06:43:28.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b756b7-2dda-43c0-b245-cc564c7cae91", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121482025-04-30T06:43:28.140ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
121492025-04-30T06:43:28.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef95858-d1b3-4182-b6c4-21cd42af264c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121502025-04-30T06:43:28.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
121512025-04-30T06:43:28.149ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c330870-6855-4c0e-a95b-ae2d42dfac6e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121522025-04-30T06:43:28.149ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
121532025-04-30T06:43:28.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72460492-b1d9-46c7-bb15-2788f5c09a6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121542025-04-30T06:43:28.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
121552025-04-30T06:43:28.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e27342-ae9a-4212-97f9-7c6d734e958e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121562025-04-30T06:43:28.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
121572025-04-30T06:43:28.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccf7a783-1d73-4b95-bf6b-e0d57bc9f77f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121582025-04-30T06:43:28.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
121592025-04-30T06:43:28.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45086818-00d8-406b-8e17-53712bb1ff0e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121602025-04-30T06:43:28.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
121612025-04-30T06:43:28.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e293094d-1b66-4a78-939d-29ec574c9f8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121622025-04-30T06:43:28.156ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121632025-04-30T06:43:28.156ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121642025-04-30T06:43:28.156ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121652025-04-30T06:43:28.157ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121662025-04-30T06:43:28.157ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121672025-04-30T06:43:28.167ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121682025-04-30T06:43:28.171ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121692025-04-30T06:43:28.171ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121702025-04-30T06:43:28.172ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121712025-04-30T06:43:28.172ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121722025-04-30T06:43:28.172ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121732025-04-30T06:43:28.176ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121742025-04-30T06:43:28.178ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
121752025-04-30T06:43:28.184ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121762025-04-30T06:43:28.189ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121772025-04-30T06:43:28.189ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121782025-04-30T06:43:28.189ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121792025-04-30T06:43:28.190ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121802025-04-30T06:43:28.300ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
121812025-04-30T06:43:28.300ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
121822025-04-30T06:43:28.301ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121832025-04-30T06:43:28.301ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121842025-04-30T06:43:28.301ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121852025-04-30T06:43:28.301ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
121862025-04-30T06:43:28.302ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121872025-04-30T06:43:28.302ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:40998 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
121882025-04-30T06:43:28.304ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:40998 req_id = 79720960-eca9-4367-8bd0-db45024d65d2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121892025-04-30T06:43:28.306ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 2739 local_addr = [::1]:45714 method = GET remote_addr = [::1]:40998 req_id = 79720960-eca9-4367-8bd0-db45024d65d2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121902025-04-30T06:43:28.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "79720960-eca9-4367-8bd0-db45024d65d2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
121912025-04-30T06:43:28.311ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121922025-04-30T06:43:28.311ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121932025-04-30T06:43:28.311ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
121942025-04-30T06:43:28.313ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
121952025-04-30T06:43:28.313ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121962025-04-30T06:43:28.313ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121972025-04-30T06:43:28.313ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121982025-04-30T06:43:28.314ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121992025-04-30T06:43:28.316ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122002025-04-30T06:43:28.322ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122012025-04-30T06:43:28.322ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122022025-04-30T06:43:28.322ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
122032025-04-30T06:43:28.323ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122042025-04-30T06:43:28.329ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122052025-04-30T06:43:28.333ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122062025-04-30T06:43:28.334ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122072025-04-30T06:43:28.334ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122082025-04-30T06:43:28.336ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122092025-04-30T06:43:28.343ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122102025-04-30T06:43:28.343ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
122112025-04-30T06:43:28.344ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122122025-04-30T06:43:28.410ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122132025-04-30T06:43:28.410ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122142025-04-30T06:43:28.410ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 703.736238ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122152025-04-30T06:43:28.410ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
122162025-04-30T06:43:28.410ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122172025-04-30T06:43:28.410ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122182025-04-30T06:43:28.410ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122192025-04-30T06:43:28.410ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122202025-04-30T06:43:28.411ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122212025-04-30T06:43:28.414ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122222025-04-30T06:43:28.421ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122232025-04-30T06:43:28.421ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122242025-04-30T06:43:28.421ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122252025-04-30T06:43:28.422ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122262025-04-30T06:43:28.428ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
122272025-04-30T06:43:28.428ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122282025-04-30T06:43:28.429ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 98 55 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .U..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60582 req_id = a3a402f1-fb5b-46fb-a9ec-b68b2ea3a769
122292025-04-30T06:43:28.429ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38997,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60582 req_id = a3a402f1-fb5b-46fb-a9ec-b68b2ea3a769
122302025-04-30T06:43:28.430ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
122312025-04-30T06:43:28.431ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122322025-04-30T06:43:28.431ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60582 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a3a402f1-fb5b-46fb-a9ec-b68b2ea3a769
122332025-04-30T06:43:28.435ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 234823789 } })
122342025-04-30T06:43:28.436ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
122352025-04-30T06:43:28.440ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
122362025-04-30T06:43:28.440ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 08 59 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62356 req_id = f79e7bb4-03f3-4d13-8a2c-58d2a8949774
122372025-04-30T06:43:28.440ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2137,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62356 req_id = f79e7bb4-03f3-4d13-8a2c-58d2a8949774
122382025-04-30T06:43:28.442ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122392025-04-30T06:43:28.443ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62356 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f79e7bb4-03f3-4d13-8a2c-58d2a8949774
122402025-04-30T06:43:28.530ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122412025-04-30T06:43:28.532ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122422025-04-30T06:43:28.533ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122432025-04-30T06:43:28.533ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122442025-04-30T06:43:28.533ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
122452025-04-30T06:43:28.534ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122462025-04-30T06:43:28.535ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
122472025-04-30T06:43:28.535ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5a 87 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Z............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55892 req_id = 34338666-5966-4f9f-8754-9dca5397a431
122482025-04-30T06:43:28.535ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23175,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55892 req_id = 34338666-5966-4f9f-8754-9dca5397a431
122492025-04-30T06:43:28.536ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
122502025-04-30T06:43:28.536ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23175, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55892 req_id = 34338666-5966-4f9f-8754-9dca5397a431
122512025-04-30T06:43:28.536ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
122522025-04-30T06:43:28.537ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
122532025-04-30T06:43:28.537ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
122542025-04-30T06:43:28.537ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c6 3c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .<..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58585 req_id = 348934be-e053-4eef-98e5-7ab44cfa2bbe
122552025-04-30T06:43:28.537ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50748,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58585 req_id = 348934be-e053-4eef-98e5-7ab44cfa2bbe
122562025-04-30T06:43:28.538ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
122572025-04-30T06:43:28.538ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122582025-04-30T06:43:28.538ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58585 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 348934be-e053-4eef-98e5-7ab44cfa2bbe
122592025-04-30T06:43:28.540ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122602025-04-30T06:43:28.550ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122612025-04-30T06:43:28.550ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122622025-04-30T06:43:28.550ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
122632025-04-30T06:43:28.552ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122642025-04-30T06:43:28.554ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 353691515 } })
122652025-04-30T06:43:28.554ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
122662025-04-30T06:43:28.558ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9b 4c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .L..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49657 req_id = 995a92d4-e0a3-4073-908e-f3622bbc14ae
122672025-04-30T06:43:28.558ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39756,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49657 req_id = 995a92d4-e0a3-4073-908e-f3622bbc14ae
122682025-04-30T06:43:28.558ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122692025-04-30T06:43:28.559ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49657 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 995a92d4-e0a3-4073-908e-f3622bbc14ae
122702025-04-30T06:43:28.559ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122712025-04-30T06:43:28.559ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
122722025-04-30T06:43:28.560ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122732025-04-30T06:43:28.561ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
122742025-04-30T06:43:28.633ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
122752025-04-30T06:43:28.633ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
122762025-04-30T06:43:28.634ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
122772025-04-30T06:43:28.634ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
122782025-04-30T06:43:28.637ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
122792025-04-30T06:43:28.641ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
122802025-04-30T06:43:28.642ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9cdf77a-4bc8-4c07-87ab-2096fdfbcf11", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:27 GMT"} })
122812025-04-30T06:43:28.642ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
122822025-04-30T06:43:28.644ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: be 30 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .0..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63526 req_id = 81da51f2-efc3-4040-a2c0-c2676255ed3b
122832025-04-30T06:43:28.644ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48688,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63526 req_id = 81da51f2-efc3-4040-a2c0-c2676255ed3b
122842025-04-30T06:43:28.645ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
122852025-04-30T06:43:28.645ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122862025-04-30T06:43:28.645ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63526 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 81da51f2-efc3-4040-a2c0-c2676255ed3b
122872025-04-30T06:43:28.647ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 446410121 } })
122882025-04-30T06:43:28.647ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
122892025-04-30T06:43:28.648ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 74 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e tt..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50088 req_id = 89f91707-7aaa-4ef2-b391-d0af9175c111
122902025-04-30T06:43:28.648ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29812,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50088 req_id = 89f91707-7aaa-4ef2-b391-d0af9175c111
122912025-04-30T06:43:28.648ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122922025-04-30T06:43:28.649ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50088 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 89f91707-7aaa-4ef2-b391-d0af9175c111
122932025-04-30T06:43:28.649ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
122942025-04-30T06:43:28.713ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
122952025-04-30T06:43:28.713ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
122962025-04-30T06:43:28.714ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
122972025-04-30T06:43:28.714ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
122982025-04-30T06:43:28.723ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
122992025-04-30T06:43:28.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a975c17c-67ec-49be-ac7d-ef74d07495e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123002025-04-30T06:43:28.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
123012025-04-30T06:43:28.725ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123022025-04-30T06:43:28.725ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123032025-04-30T06:43:28.728ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123042025-04-30T06:43:28.730ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123052025-04-30T06:43:28.730ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123062025-04-30T06:43:28.730ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123072025-04-30T06:43:28.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a6c888-cd12-4455-bf7b-d8fa0f44a313", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123082025-04-30T06:43:28.739ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
123092025-04-30T06:43:28.745ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af83ee38-9295-45b7-832a-ea9802a1ef60", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123102025-04-30T06:43:28.747ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
123112025-04-30T06:43:28.754ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
123122025-04-30T06:43:28.757ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b85c9aaa-68ea-47d0-8545-1fcf17083dd8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123132025-04-30T06:43:28.758ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
123142025-04-30T06:43:28.763ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e544dd14-8d7f-4738-9043-61ef0951c550", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123152025-04-30T06:43:28.765ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
123162025-04-30T06:43:28.767ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b60fae97-b158-4f77-94e4-86ba6354f7dd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123172025-04-30T06:43:28.768ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
123182025-04-30T06:43:28.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8416181d-4e12-4374-9317-f69f6308e82a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123192025-04-30T06:43:28.771ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
123202025-04-30T06:43:28.772ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57a5088-a0b5-4fe8-a6f5-4a96d18dfba2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123212025-04-30T06:43:28.772ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
123222025-04-30T06:43:28.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9af7309-78da-42b0-9cf1-f1b288dacd64", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123232025-04-30T06:43:28.773ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
123242025-04-30T06:43:28.774ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52a8d76e-e26d-4994-a2f5-ac88a185d71a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123252025-04-30T06:43:28.775ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
123262025-04-30T06:43:28.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ab3238-5976-4bde-81d5-a99c9eb6377e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123272025-04-30T06:43:28.781ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
123282025-04-30T06:43:28.786ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7398bcbb-041d-4e96-9435-58cb06746d73", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123292025-04-30T06:43:28.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
123302025-04-30T06:43:28.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fe81066-84ed-4128-aa55-6911dc06b633", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123312025-04-30T06:43:28.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
123322025-04-30T06:43:28.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db3cb221-6b61-4bbf-b4f3-7f42f42b6a45", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123332025-04-30T06:43:28.796ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
123342025-04-30T06:43:28.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50ef9331-1cdc-4617-b732-79f2b7d911ac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123352025-04-30T06:43:28.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
123362025-04-30T06:43:28.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b39ad2-09a6-4f91-9388-3ea2bf25ce88", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123372025-04-30T06:43:28.803ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
123382025-04-30T06:43:28.806ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123392025-04-30T06:43:28.806ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123402025-04-30T06:43:28.806ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123412025-04-30T06:43:28.806ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
123422025-04-30T06:43:28.807ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123432025-04-30T06:43:28.808ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52c6db66-8854-45ae-8672-b56803606d2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123442025-04-30T06:43:28.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
123452025-04-30T06:43:28.810ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123462025-04-30T06:43:28.817ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123472025-04-30T06:43:28.817ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123482025-04-30T06:43:28.817ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
123492025-04-30T06:43:28.817ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123502025-04-30T06:43:28.821ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123512025-04-30T06:43:28.822ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
123522025-04-30T06:43:28.822ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123532025-04-30T06:43:28.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4203da57-7b11-4965-9c2a-d61a764b18d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123542025-04-30T06:43:28.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
123552025-04-30T06:43:28.825ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e2605b-aa26-4765-b340-fd3c9e3503a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123562025-04-30T06:43:28.825ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
123572025-04-30T06:43:28.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6684e9a3-6d4e-40c3-950c-19074582912d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123582025-04-30T06:43:28.828ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
123592025-04-30T06:43:28.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12fcf444-c874-4094-a0cb-fb35c1bd1d87", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123602025-04-30T06:43:28.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
123612025-04-30T06:43:28.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33fcafe7-ce86-49bd-80c1-0350faa64ec2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123622025-04-30T06:43:28.831ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
123632025-04-30T06:43:28.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4b2f0d-68ea-49ee-a1c8-11f076660110", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123642025-04-30T06:43:28.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
123652025-04-30T06:43:28.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c23e799-e81b-498d-a879-362f4626dbfd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123662025-04-30T06:43:28.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
123672025-04-30T06:43:28.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb83d2f-6174-4055-a2d1-43c6ec08956b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123682025-04-30T06:43:28.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
123692025-04-30T06:43:28.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b98dd42-afa2-41f9-9ecd-bddde9368662", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123702025-04-30T06:43:28.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
123712025-04-30T06:43:28.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41dd82f9-fada-4b89-aacb-ec042ad2d8ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123722025-04-30T06:43:28.846ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
123732025-04-30T06:43:28.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f100941f-e476-4c81-8277-4ce5ea84766f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123742025-04-30T06:43:28.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
123752025-04-30T06:43:28.853ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
123762025-04-30T06:43:28.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68d1b1d0-d35f-402c-9477-d04947dc667b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123772025-04-30T06:43:28.854ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
123782025-04-30T06:43:28.855ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f27bf39-5105-416e-9745-9474159646af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123792025-04-30T06:43:28.855ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
123802025-04-30T06:43:28.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17c52ceb-4d1d-48c9-9ebe-875e42251d87", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123812025-04-30T06:43:28.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
123822025-04-30T06:43:28.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e20ecc-fdb0-4a78-99ab-c7c6cc2ff30e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123832025-04-30T06:43:28.859ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
123842025-04-30T06:43:28.860ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "775fc69e-ee93-48d5-8d4c-a7eaf1f2a9fe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123852025-04-30T06:43:28.860ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
123862025-04-30T06:43:28.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69de4087-b4c0-482c-b040-bdf4c28f6b7c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123872025-04-30T06:43:28.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
123882025-04-30T06:43:28.863ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "644d2052-54f5-4b02-afb4-6516d8baa045", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123892025-04-30T06:43:28.863ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
123902025-04-30T06:43:28.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6376b6f-6ad0-4f0d-a5b7-e51146d446ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123912025-04-30T06:43:28.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
123922025-04-30T06:43:28.869ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a57a3224-22d4-4d41-929a-88b09275bf78", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123932025-04-30T06:43:28.869ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
123942025-04-30T06:43:28.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5013f68d-d5e6-4d6b-b036-8cadf95d5b46", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123952025-04-30T06:43:28.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
123962025-04-30T06:43:28.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c366814d-61d8-42dd-a383-5d7ff476deb9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123972025-04-30T06:43:28.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
123982025-04-30T06:43:28.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bce2a309-ff23-4a21-aa76-893c153b2c5e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
123992025-04-30T06:43:28.875ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
124002025-04-30T06:43:28.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78c4029a-9775-4882-bd75-6b5d5348c382", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
124012025-04-30T06:43:28.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
124022025-04-30T06:43:28.877ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fef9c6b-1464-404a-87ab-caebbced835a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
124032025-04-30T06:43:28.877ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
124042025-04-30T06:43:28.878ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56471923-0303-4033-8523-c817cad65fec", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
124052025-04-30T06:43:28.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
124062025-04-30T06:43:28.880ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d488e2-6659-4ece-b565-5bb6cf23d354", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
124072025-04-30T06:43:28.880ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124082025-04-30T06:43:28.880ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124092025-04-30T06:43:28.880ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124102025-04-30T06:43:28.880ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124112025-04-30T06:43:28.882ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124122025-04-30T06:43:28.889ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124132025-04-30T06:43:28.893ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124142025-04-30T06:43:28.893ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124152025-04-30T06:43:28.893ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124162025-04-30T06:43:28.893ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124172025-04-30T06:43:28.893ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124182025-04-30T06:43:28.895ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124192025-04-30T06:43:28.898ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124202025-04-30T06:43:28.902ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124212025-04-30T06:43:28.902ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124222025-04-30T06:43:28.902ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124232025-04-30T06:43:28.903ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124242025-04-30T06:43:28.904ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124252025-04-30T06:43:28.904ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124262025-04-30T06:43:28.904ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124272025-04-30T06:43:28.905ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
124282025-04-30T06:43:28.905ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124292025-04-30T06:43:28.908ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124302025-04-30T06:43:28.913ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124312025-04-30T06:43:28.913ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124322025-04-30T06:43:28.913ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
124332025-04-30T06:43:28.914ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124342025-04-30T06:43:28.993ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
124352025-04-30T06:43:28.993ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
124362025-04-30T06:43:28.997ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124372025-04-30T06:43:28.997ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
124382025-04-30T06:43:28.998ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124392025-04-30T06:43:28.998ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:54643 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124402025-04-30T06:43:28.998ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124412025-04-30T06:43:28.998ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124422025-04-30T06:43:28.999ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
124432025-04-30T06:43:29.003ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
124442025-04-30T06:43:29.003ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:54643 req_id = 97968e4f-4fc6-47d6-aadf-265f4aea5296 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124452025-04-30T06:43:29.004ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 625 local_addr = [::1]:45714 method = GET remote_addr = [::1]:54643 req_id = 97968e4f-4fc6-47d6-aadf-265f4aea5296 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124462025-04-30T06:43:29.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "97968e4f-4fc6-47d6-aadf-265f4aea5296", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
124472025-04-30T06:43:29.010ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124482025-04-30T06:43:29.010ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124492025-04-30T06:43:29.010ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
124502025-04-30T06:43:29.012ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124512025-04-30T06:43:29.012ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124522025-04-30T06:43:29.012ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124532025-04-30T06:43:29.012ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124542025-04-30T06:43:29.013ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124552025-04-30T06:43:29.014ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
124562025-04-30T06:43:29.014ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
124572025-04-30T06:43:29.016ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124582025-04-30T06:43:29.020ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124592025-04-30T06:43:29.020ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124602025-04-30T06:43:29.021ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124612025-04-30T06:43:29.022ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124622025-04-30T06:43:29.023ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
124632025-04-30T06:43:29.023ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124642025-04-30T06:43:29.050ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
124652025-04-30T06:43:29.067ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
124662025-04-30T06:43:29.068ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
124672025-04-30T06:43:29.068ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 656.974889ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
124682025-04-30T06:43:29.068ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
124692025-04-30T06:43:29.068ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124702025-04-30T06:43:29.068ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124712025-04-30T06:43:29.068ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124722025-04-30T06:43:29.068ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124732025-04-30T06:43:29.068ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124742025-04-30T06:43:29.073ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124752025-04-30T06:43:29.079ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124762025-04-30T06:43:29.079ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124772025-04-30T06:43:29.079ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124782025-04-30T06:43:29.080ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124792025-04-30T06:43:29.084ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
124802025-04-30T06:43:29.084ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
124812025-04-30T06:43:29.086ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c9 ac 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55975 req_id = 16ccd1af-af53-4e97-80a3-1c9f8152eb94
124822025-04-30T06:43:29.086ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51628,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55975 req_id = 16ccd1af-af53-4e97-80a3-1c9f8152eb94
124832025-04-30T06:43:29.086ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
124842025-04-30T06:43:29.086ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124852025-04-30T06:43:29.086ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55975 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 16ccd1af-af53-4e97-80a3-1c9f8152eb94
124862025-04-30T06:43:29.087ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4273, tv_nsec: 885949484 } })
124872025-04-30T06:43:29.087ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
124882025-04-30T06:43:29.087ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 e0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63141 req_id = 33a3572d-cdfe-4ca0-8945-8780729c3a7b
124892025-04-30T06:43:29.088ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2528,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63141 req_id = 33a3572d-cdfe-4ca0-8945-8780729c3a7b
124902025-04-30T06:43:29.088ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124912025-04-30T06:43:29.088ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63141 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 33a3572d-cdfe-4ca0-8945-8780729c3a7b
124922025-04-30T06:43:29.169ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
124932025-04-30T06:43:29.170ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124942025-04-30T06:43:29.170ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124952025-04-30T06:43:29.170ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124962025-04-30T06:43:29.170ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
124972025-04-30T06:43:29.170ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124982025-04-30T06:43:29.171ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
124992025-04-30T06:43:29.171ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 77 67 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c wg..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57832 req_id = c8057454-9b60-4fa0-8d53-91c19a9d80d0
125002025-04-30T06:43:29.171ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30567,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57832 req_id = c8057454-9b60-4fa0-8d53-91c19a9d80d0
125012025-04-30T06:43:29.171ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
125022025-04-30T06:43:29.171ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30567, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57832 req_id = c8057454-9b60-4fa0-8d53-91c19a9d80d0
125032025-04-30T06:43:29.171ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
125042025-04-30T06:43:29.171ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
125052025-04-30T06:43:29.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
125062025-04-30T06:43:29.174ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125072025-04-30T06:43:29.179ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125082025-04-30T06:43:29.179ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125092025-04-30T06:43:29.179ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
125102025-04-30T06:43:29.179ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125112025-04-30T06:43:29.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c124a8-e1d4-4abe-8136-79453c6cbe07", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125122025-04-30T06:43:29.180ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
125132025-04-30T06:43:29.182ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125142025-04-30T06:43:29.183ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
125152025-04-30T06:43:29.183ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125162025-04-30T06:43:29.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e1a0cb-02db-4931-904c-351409e5bcbc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125172025-04-30T06:43:29.184ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
125182025-04-30T06:43:29.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe06bf7a-5b3d-44e4-b80a-d6b4909196b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125192025-04-30T06:43:29.186ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
125202025-04-30T06:43:29.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed9cd91-a7a7-4b4b-bde6-93418c93120e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125212025-04-30T06:43:29.189ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
125222025-04-30T06:43:29.192ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3add2b4f-c5ec-45ce-bec7-2c2680c442c6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125232025-04-30T06:43:29.192ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
125242025-04-30T06:43:29.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b34f776-252a-4192-9d9d-861eab4a5f89", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125252025-04-30T06:43:29.195ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
125262025-04-30T06:43:29.199ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61e51e41-67f7-4408-9d2a-32986ca5a281", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125272025-04-30T06:43:29.199ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
125282025-04-30T06:43:29.202ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06856675-6eb8-49e9-8839-c8da3d82d0e1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125292025-04-30T06:43:29.202ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
125302025-04-30T06:43:29.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39fc1fc0-b244-4e13-bf92-134c1ea7402c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125312025-04-30T06:43:29.204ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
125322025-04-30T06:43:29.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65aab0ac-865a-49bf-9eb3-ca803198e559", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125332025-04-30T06:43:29.205ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
125342025-04-30T06:43:29.206ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c599bdb9-5155-4b16-8507-c3514693ddb5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125352025-04-30T06:43:29.207ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
125362025-04-30T06:43:29.208ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "480973ff-9d96-40b1-ae37-76448f48d3aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125372025-04-30T06:43:29.208ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
125382025-04-30T06:43:29.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7e8a4ab-716d-4941-81e2-36ff1be4e4e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125392025-04-30T06:43:29.213ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
125402025-04-30T06:43:29.215ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
125412025-04-30T06:43:29.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3b4ec54-165b-4c76-9a93-6c954b3c5be5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125422025-04-30T06:43:29.216ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
125432025-04-30T06:43:29.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1b3ceaa-aa8f-427e-a0d4-04375b587a50", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125442025-04-30T06:43:29.218ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
125452025-04-30T06:43:29.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1c8a0b7-741d-49b9-9a38-5f0036d7017c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125462025-04-30T06:43:29.221ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
125472025-04-30T06:43:29.225ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b3a01b-788b-46f4-983f-a8845719031a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125482025-04-30T06:43:29.226ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
125492025-04-30T06:43:29.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e2da920-a2d6-4384-aad0-c9d2970c29da", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125502025-04-30T06:43:29.230ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
125512025-04-30T06:43:29.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bffa979-af67-4757-b5f9-3d14d6071b0d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125522025-04-30T06:43:29.240ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
125532025-04-30T06:43:29.245ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7177c92a-aefc-4d4a-86b1-2c771beb2379", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125542025-04-30T06:43:29.246ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
125552025-04-30T06:43:29.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbcfc1c8-d711-4787-9a31-39cb8deff53a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125562025-04-30T06:43:29.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
125572025-04-30T06:43:29.250ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cd9e4c9-781c-4221-85e4-d7c7021ec8bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125582025-04-30T06:43:29.251ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
125592025-04-30T06:43:29.263ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b905a770-6711-4dec-90f5-a61af47bd643", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125602025-04-30T06:43:29.264ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
125612025-04-30T06:43:29.273ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125622025-04-30T06:43:29.273ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125632025-04-30T06:43:29.273ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125642025-04-30T06:43:29.273ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
125652025-04-30T06:43:29.274ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125662025-04-30T06:43:29.279ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125672025-04-30T06:43:29.284ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125682025-04-30T06:43:29.284ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125692025-04-30T06:43:29.284ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
125702025-04-30T06:43:29.284ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125712025-04-30T06:43:29.289ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125722025-04-30T06:43:29.289ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
125732025-04-30T06:43:29.291ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125742025-04-30T06:43:29.295ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50d9c514-1c19-41f6-9368-c1010272882e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125752025-04-30T06:43:29.297ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
125762025-04-30T06:43:29.334ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "786143e8-469e-4828-8e85-a13d6b6ec644", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125772025-04-30T06:43:29.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
125782025-04-30T06:43:29.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8ede24f-b248-43a9-83f7-4fe4e8c3ec83", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125792025-04-30T06:43:29.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
125802025-04-30T06:43:29.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26e6766b-53a1-4a26-bb06-48958a84cbed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125812025-04-30T06:43:29.343ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
125822025-04-30T06:43:29.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6085b03-729a-4254-85f7-d2d0cda573dc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125832025-04-30T06:43:29.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
125842025-04-30T06:43:29.352ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6fd5726-bc9d-4e8a-8353-317453ccffbd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125852025-04-30T06:43:29.352ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
125862025-04-30T06:43:29.353ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
125872025-04-30T06:43:29.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42547966-1010-4a3a-b929-8279b7c7c48e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125882025-04-30T06:43:29.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
125892025-04-30T06:43:29.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f8f361-a1cd-4c81-9fd8-43af3cf23dc0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125902025-04-30T06:43:29.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
125912025-04-30T06:43:29.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a0c92c6-dff0-4dd7-8029-1733e417c0a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125922025-04-30T06:43:29.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
125932025-04-30T06:43:29.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc2f464b-8737-4457-a6e2-796e283ef3f3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125942025-04-30T06:43:29.367ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
125952025-04-30T06:43:29.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cda708e-a5e1-4cc2-b4c8-1033c935ae17", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125962025-04-30T06:43:29.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
125972025-04-30T06:43:29.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a8efde2-ad13-4a57-82fa-b6410cae4266", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
125982025-04-30T06:43:29.370ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
125992025-04-30T06:43:29.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93569f85-0603-4365-bbaf-d5484e2798c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126002025-04-30T06:43:29.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
126012025-04-30T06:43:29.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc963550-e9d7-467c-a36d-e06dce3f8188", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126022025-04-30T06:43:29.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
126032025-04-30T06:43:29.377ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ba8201f-d983-4a1c-801e-c3a99b0db1b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126042025-04-30T06:43:29.377ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
126052025-04-30T06:43:29.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce980a83-a6cb-43d3-91ef-27833d96adb9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126062025-04-30T06:43:29.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
126072025-04-30T06:43:29.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f664db74-b4b3-4fc7-be87-d5c1b3f6e9d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126082025-04-30T06:43:29.382ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
126092025-04-30T06:43:29.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b857652b-5ef6-4b56-9588-4565a8259108", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126102025-04-30T06:43:29.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
126112025-04-30T06:43:29.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0e3364d-8082-4cce-a79a-48ceac1299b2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126122025-04-30T06:43:29.385ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
126132025-04-30T06:43:29.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa177ed4-e49d-4863-837b-b45fd28dd1cf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126142025-04-30T06:43:29.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
126152025-04-30T06:43:29.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce83c52-96cc-436c-a1d2-52096920a56d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126162025-04-30T06:43:29.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
126172025-04-30T06:43:29.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f399484-36a9-418b-a8d1-989834e15c5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126182025-04-30T06:43:29.389ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126192025-04-30T06:43:29.389ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126202025-04-30T06:43:29.389ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126212025-04-30T06:43:29.389ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126222025-04-30T06:43:29.394ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126232025-04-30T06:43:29.400ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126242025-04-30T06:43:29.413ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126252025-04-30T06:43:29.413ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126262025-04-30T06:43:29.413ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126272025-04-30T06:43:29.413ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126282025-04-30T06:43:29.413ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126292025-04-30T06:43:29.414ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126302025-04-30T06:43:29.414ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126312025-04-30T06:43:29.415ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126322025-04-30T06:43:29.415ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126332025-04-30T06:43:29.415ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
126342025-04-30T06:43:29.415ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126352025-04-30T06:43:29.419ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126362025-04-30T06:43:29.430ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126372025-04-30T06:43:29.430ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126382025-04-30T06:43:29.432ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
126392025-04-30T06:43:29.432ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126402025-04-30T06:43:29.432ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126412025-04-30T06:43:29.455ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126422025-04-30T06:43:29.455ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126432025-04-30T06:43:29.456ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126442025-04-30T06:43:29.463ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126452025-04-30T06:43:29.472ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126462025-04-30T06:43:29.472ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
126472025-04-30T06:43:29.473ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126482025-04-30T06:43:29.567ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
126492025-04-30T06:43:29.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
126502025-04-30T06:43:29.571ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:51194 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126512025-04-30T06:43:29.572ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:51194 req_id = aad92ec0-eed6-40dd-bbac-dd37776aa8bf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126522025-04-30T06:43:29.573ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 638 local_addr = [::1]:45714 method = GET remote_addr = [::1]:51194 req_id = aad92ec0-eed6-40dd-bbac-dd37776aa8bf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126532025-04-30T06:43:29.573ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "aad92ec0-eed6-40dd-bbac-dd37776aa8bf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126542025-04-30T06:43:29.578ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126552025-04-30T06:43:29.579ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126562025-04-30T06:43:29.579ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
126572025-04-30T06:43:29.581ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126582025-04-30T06:43:29.581ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126592025-04-30T06:43:29.581ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126602025-04-30T06:43:29.581ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126612025-04-30T06:43:29.583ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126622025-04-30T06:43:29.589ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126632025-04-30T06:43:29.598ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126642025-04-30T06:43:29.599ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126652025-04-30T06:43:29.599ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126662025-04-30T06:43:29.600ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126672025-04-30T06:43:29.635ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
126682025-04-30T06:43:29.636ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126692025-04-30T06:43:29.636ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = fb719b83-dc9c-4d9c-948f-82d29b4f8b3c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126702025-04-30T06:43:29.637ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = fb719b83-dc9c-4d9c-948f-82d29b4f8b3c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126712025-04-30T06:43:29.637ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = fb719b83-dc9c-4d9c-948f-82d29b4f8b3c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126722025-04-30T06:43:29.638ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = fb719b83-dc9c-4d9c-948f-82d29b4f8b3c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126732025-04-30T06:43:29.642ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 5517 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = fb719b83-dc9c-4d9c-948f-82d29b4f8b3c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
126742025-04-30T06:43:29.646ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb719b83-dc9c-4d9c-948f-82d29b4f8b3c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:28 GMT"} })
126752025-04-30T06:43:29.646ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126762025-04-30T06:43:29.647ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 20f58c43-a717-4ef4-acdf-4d9a2b7bfac7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126772025-04-30T06:43:29.648ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 20f58c43-a717-4ef4-acdf-4d9a2b7bfac7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126782025-04-30T06:43:29.648ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 20f58c43-a717-4ef4-acdf-4d9a2b7bfac7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126792025-04-30T06:43:29.650ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 20f58c43-a717-4ef4-acdf-4d9a2b7bfac7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126802025-04-30T06:43:29.656ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
126812025-04-30T06:43:29.657ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 10376 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 20f58c43-a717-4ef4-acdf-4d9a2b7bfac7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
126822025-04-30T06:43:29.658ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "20f58c43-a717-4ef4-acdf-4d9a2b7bfac7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
126832025-04-30T06:43:29.658ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126842025-04-30T06:43:29.658ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
126852025-04-30T06:43:29.658ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
126862025-04-30T06:43:29.670ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
126872025-04-30T06:43:29.671ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
126882025-04-30T06:43:29.671ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 603.545105ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
126892025-04-30T06:43:29.671ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
126902025-04-30T06:43:29.671ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126912025-04-30T06:43:29.671ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126922025-04-30T06:43:29.671ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126932025-04-30T06:43:29.671ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126942025-04-30T06:43:29.672ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126952025-04-30T06:43:29.676ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126962025-04-30T06:43:29.692ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126972025-04-30T06:43:29.692ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126982025-04-30T06:43:29.692ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126992025-04-30T06:43:29.693ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127002025-04-30T06:43:29.697ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
127012025-04-30T06:43:29.697ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
127022025-04-30T06:43:29.698ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ab a1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49506 req_id = 18190c6e-384c-40ba-be2d-babd78564a8d
127032025-04-30T06:43:29.698ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43937,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49506 req_id = 18190c6e-384c-40ba-be2d-babd78564a8d
127042025-04-30T06:43:29.698ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
127052025-04-30T06:43:29.698ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127062025-04-30T06:43:29.698ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49506 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 18190c6e-384c-40ba-be2d-babd78564a8d
127072025-04-30T06:43:29.699ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4274, tv_nsec: 498026398 } })
127082025-04-30T06:43:29.699ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
127092025-04-30T06:43:29.699ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58722 req_id = cbcb99a7-46da-4661-a377-bf3407344d12
127102025-04-30T06:43:29.699ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33817,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58722 req_id = cbcb99a7-46da-4661-a377-bf3407344d12
127112025-04-30T06:43:29.699ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127122025-04-30T06:43:29.699ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58722 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cbcb99a7-46da-4661-a377-bf3407344d12
127132025-04-30T06:43:29.764ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127142025-04-30T06:43:29.764ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127152025-04-30T06:43:29.764ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127162025-04-30T06:43:29.765ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127172025-04-30T06:43:29.765ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
127182025-04-30T06:43:29.766ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127192025-04-30T06:43:29.766ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127202025-04-30T06:43:29.766ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127212025-04-30T06:43:29.767ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 65 da 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c e............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55329 req_id = 4f520e3d-b700-43d2-b7e1-37f21c1779cc
127222025-04-30T06:43:29.767ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26074,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55329 req_id = 4f520e3d-b700-43d2-b7e1-37f21c1779cc
127232025-04-30T06:43:29.768ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
127242025-04-30T06:43:29.768ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55329 req_id = 4f520e3d-b700-43d2-b7e1-37f21c1779cc
127252025-04-30T06:43:29.770ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
127262025-04-30T06:43:29.770ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
127272025-04-30T06:43:29.770ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
127282025-04-30T06:43:29.770ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127292025-04-30T06:43:29.771ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127302025-04-30T06:43:29.771ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127312025-04-30T06:43:29.771ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127322025-04-30T06:43:29.772ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127332025-04-30T06:43:29.777ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127342025-04-30T06:43:29.777ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127352025-04-30T06:43:29.777ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
127362025-04-30T06:43:29.777ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127372025-04-30T06:43:29.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b7aeaf-fe7d-48cf-928c-b0cb360e0318", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127382025-04-30T06:43:29.780ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
127392025-04-30T06:43:29.781ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127402025-04-30T06:43:29.781ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
127412025-04-30T06:43:29.782ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127422025-04-30T06:43:29.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54f4271c-aa9a-49c5-96c7-2830c7aa7eda", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127432025-04-30T06:43:29.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
127442025-04-30T06:43:29.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96507beb-d684-44bf-9406-2a28b6626137", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127452025-04-30T06:43:29.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
127462025-04-30T06:43:29.789ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f4cc3cf-c524-4524-a28a-a5c39f190cc3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127472025-04-30T06:43:29.790ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
127482025-04-30T06:43:29.794ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e5a292-cda6-46cf-83c3-ce51f742eced", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127492025-04-30T06:43:29.794ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
127502025-04-30T06:43:29.798ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c2be05d-c74d-4697-af43-bd2ea5579c34", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127512025-04-30T06:43:29.799ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
127522025-04-30T06:43:29.812ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f73ef49-abe9-4846-ae40-78857ba5bf1b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127532025-04-30T06:43:29.812ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
127542025-04-30T06:43:29.819ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
127552025-04-30T06:43:29.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f13ed3d3-2447-46b9-bcf9-aa41ab0da213", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127562025-04-30T06:43:29.822ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
127572025-04-30T06:43:29.823ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "916336f1-10e1-4767-bcec-0f27af1a46a8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127582025-04-30T06:43:29.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
127592025-04-30T06:43:29.825ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29dee521-195e-4a15-9989-8ea5f37612df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127602025-04-30T06:43:29.826ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
127612025-04-30T06:43:29.827ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07b03aaa-8b70-4291-80a2-7bf7474dd901", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127622025-04-30T06:43:29.827ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
127632025-04-30T06:43:29.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8233d8c-68fb-48ea-b8d7-be6ee47d2b78", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127642025-04-30T06:43:29.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
127652025-04-30T06:43:29.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "352cce60-cebf-4a6d-a931-44d4409d333b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127662025-04-30T06:43:29.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
127672025-04-30T06:43:29.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9afd3ec7-c952-4fc7-a01c-e48b0102b817", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127682025-04-30T06:43:29.832ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
127692025-04-30T06:43:29.836ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3183e994-33c8-4f27-bc47-6c86b475eeef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127702025-04-30T06:43:29.837ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
127712025-04-30T06:43:29.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e3c32ab-ff15-4d77-8a0f-eaecfcddfb7d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127722025-04-30T06:43:29.839ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
127732025-04-30T06:43:29.841ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f73205f-15f6-4ea1-8c6c-a3ac6462ab32", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127742025-04-30T06:43:29.841ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
127752025-04-30T06:43:29.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "833ebe10-e465-495e-b1f7-8b9909f7b8fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127762025-04-30T06:43:29.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
127772025-04-30T06:43:29.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "326d6d83-25d0-49bb-8b51-5bd59b8c7bc1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127782025-04-30T06:43:29.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
127792025-04-30T06:43:29.850ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6816442c-ad63-40d2-85aa-b118b0c9bb5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127802025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
127812025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c17de83-7baa-4f60-8d95-bfe83b90102a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127822025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
127832025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "264e97f0-6090-43df-ad13-3e1dd93d06d2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127842025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
127852025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1b92679-957d-4e5e-a77e-167bea83c34c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127862025-04-30T06:43:29.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
127872025-04-30T06:43:29.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "458c175e-4a08-46c5-97ea-3a82a62d81a3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127882025-04-30T06:43:29.856ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
127892025-04-30T06:43:29.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6bcaaa-34a9-4219-bdc4-e5a40dc0c0e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127902025-04-30T06:43:29.858ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
127912025-04-30T06:43:29.860ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cb9833a-3933-4c17-aad1-ab1e89a9c373", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127922025-04-30T06:43:29.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
127932025-04-30T06:43:29.864ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b18446-0341-48b7-8ca5-ea54350b583b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127942025-04-30T06:43:29.864ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
127952025-04-30T06:43:29.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de502c9-f34b-4b92-a495-5c959e44fb99", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
127962025-04-30T06:43:29.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
127972025-04-30T06:43:29.871ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127982025-04-30T06:43:29.871ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127992025-04-30T06:43:29.871ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128002025-04-30T06:43:29.871ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
128012025-04-30T06:43:29.872ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128022025-04-30T06:43:29.877ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6540ed4b-d7b2-4467-b5d8-681a7ab051c3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128032025-04-30T06:43:29.877ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
128042025-04-30T06:43:29.877ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128052025-04-30T06:43:29.886ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128062025-04-30T06:43:29.886ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128072025-04-30T06:43:29.886ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
128082025-04-30T06:43:29.887ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128092025-04-30T06:43:29.889ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128102025-04-30T06:43:29.889ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
128112025-04-30T06:43:29.890ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128122025-04-30T06:43:29.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5401676d-ba50-411d-b5ed-b41f38d0835a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128132025-04-30T06:43:29.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
128142025-04-30T06:43:29.892ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df134ae1-99c3-4ea5-9a49-254418922296", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128152025-04-30T06:43:29.892ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
128162025-04-30T06:43:29.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f75f1e5-7f42-4663-9bb1-be6327413fc5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128172025-04-30T06:43:29.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
128182025-04-30T06:43:29.897ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "955e3c5b-7538-4368-914b-bc95729e5844", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128192025-04-30T06:43:29.897ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
128202025-04-30T06:43:29.898ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ee3d279-397a-48b5-8a0b-f1708ef81a0c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128212025-04-30T06:43:29.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
128222025-04-30T06:43:29.907ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "620616a2-e75c-4639-a138-f7ae61aff52a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128232025-04-30T06:43:29.907ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
128242025-04-30T06:43:29.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e975ff49-2625-4cbf-855c-775d0010fa1d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128252025-04-30T06:43:29.911ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
128262025-04-30T06:43:29.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a2d316-69ee-485a-b7ae-2cc8f945e655", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128272025-04-30T06:43:29.915ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
128282025-04-30T06:43:29.917ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68b069d-3298-4e4b-8c5c-74d71118855a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128292025-04-30T06:43:29.917ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
128302025-04-30T06:43:29.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e30ea2ac-63f8-4061-b8fb-5e9d5f047da0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128312025-04-30T06:43:29.920ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
128322025-04-30T06:43:29.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48625f9-ecfe-4b5c-9aea-2a54d89dbf4f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128332025-04-30T06:43:29.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
128342025-04-30T06:43:29.926ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
128352025-04-30T06:43:29.930ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6edec09f-9f9f-4e12-9fb6-eeb62f502e73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128362025-04-30T06:43:29.930ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
128372025-04-30T06:43:29.932ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "729ae72a-8524-49fa-8ce8-0e747b5fd94f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128382025-04-30T06:43:29.932ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
128392025-04-30T06:43:29.933ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5908aae-68b7-43c1-89d6-32423d5a49da", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128402025-04-30T06:43:29.933ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
128412025-04-30T06:43:29.934ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65cf0d7e-45cb-432c-9c02-1257d76e9594", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128422025-04-30T06:43:29.935ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
128432025-04-30T06:43:29.936ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddaf5876-1bc5-4328-9556-abec957812bd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128442025-04-30T06:43:29.936ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128452025-04-30T06:43:29.936ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128462025-04-30T06:43:29.936ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128472025-04-30T06:43:29.937ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128482025-04-30T06:43:29.937ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128492025-04-30T06:43:29.941ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128502025-04-30T06:43:29.948ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128512025-04-30T06:43:29.948ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128522025-04-30T06:43:29.948ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128532025-04-30T06:43:29.948ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128542025-04-30T06:43:29.948ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128552025-04-30T06:43:29.948ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128562025-04-30T06:43:29.952ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128572025-04-30T06:43:29.957ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128582025-04-30T06:43:29.960ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128592025-04-30T06:43:29.960ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128602025-04-30T06:43:29.960ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128612025-04-30T06:43:30.055ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
128622025-04-30T06:43:30.055ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
128632025-04-30T06:43:30.056ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128642025-04-30T06:43:30.056ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128652025-04-30T06:43:30.056ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128662025-04-30T06:43:30.056ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
128672025-04-30T06:43:30.057ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128682025-04-30T06:43:30.057ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:50119 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128692025-04-30T06:43:30.058ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128702025-04-30T06:43:30.058ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128712025-04-30T06:43:30.058ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
128722025-04-30T06:43:30.060ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
128732025-04-30T06:43:30.061ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
128742025-04-30T06:43:30.061ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:50119 req_id = 9bf19652-7e28-4301-83a2-e1079721705a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128752025-04-30T06:43:30.062ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 550 local_addr = [::1]:45714 method = GET remote_addr = [::1]:50119 req_id = 9bf19652-7e28-4301-83a2-e1079721705a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128762025-04-30T06:43:30.062ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: bd aa 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62117 req_id = 54a54a36-c507-4780-b6f3-0ae56bce92bb
128772025-04-30T06:43:30.063ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48554,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62117 req_id = 54a54a36-c507-4780-b6f3-0ae56bce92bb
128782025-04-30T06:43:30.063ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
128792025-04-30T06:43:30.063ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
128802025-04-30T06:43:30.063ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62117 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 54a54a36-c507-4780-b6f3-0ae56bce92bb
128812025-04-30T06:43:30.063ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bf19652-7e28-4301-83a2-e1079721705a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
128822025-04-30T06:43:30.077ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128832025-04-30T06:43:30.077ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128842025-04-30T06:43:30.077ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
128852025-04-30T06:43:30.081ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128862025-04-30T06:43:30.081ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128872025-04-30T06:43:30.081ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128882025-04-30T06:43:30.081ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128892025-04-30T06:43:30.082ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128902025-04-30T06:43:30.083ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128912025-04-30T06:43:30.092ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128922025-04-30T06:43:30.092ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128932025-04-30T06:43:30.092ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
128942025-04-30T06:43:30.093ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128952025-04-30T06:43:30.094ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4274, tv_nsec: 893466717 } })
128962025-04-30T06:43:30.094ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
128972025-04-30T06:43:30.096ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6d 79 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e my..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50383 req_id = 0dea1008-3787-4ef8-a2df-872895309e7e
128982025-04-30T06:43:30.096ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28025,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50383 req_id = 0dea1008-3787-4ef8-a2df-872895309e7e
128992025-04-30T06:43:30.096ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129002025-04-30T06:43:30.096ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50383 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0dea1008-3787-4ef8-a2df-872895309e7e
129012025-04-30T06:43:30.096ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129022025-04-30T06:43:30.104ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129032025-04-30T06:43:30.104ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129042025-04-30T06:43:30.104ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129052025-04-30T06:43:30.104ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129062025-04-30T06:43:30.109ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129072025-04-30T06:43:30.173ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
129082025-04-30T06:43:30.174ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
129092025-04-30T06:43:30.174ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
129102025-04-30T06:43:30.174ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
129112025-04-30T06:43:30.176ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
129122025-04-30T06:43:30.177ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
129132025-04-30T06:43:30.177ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
129142025-04-30T06:43:30.178ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129152025-04-30T06:43:30.178ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
129162025-04-30T06:43:30.179ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129172025-04-30T06:43:30.179ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
129182025-04-30T06:43:30.184ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
129192025-04-30T06:43:30.184ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129202025-04-30T06:43:30.214ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
129212025-04-30T06:43:30.215ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
129222025-04-30T06:43:30.215ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 544.083621ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
129232025-04-30T06:43:30.215ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
129242025-04-30T06:43:30.215ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129252025-04-30T06:43:30.215ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129262025-04-30T06:43:30.215ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129272025-04-30T06:43:30.215ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129282025-04-30T06:43:30.216ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129292025-04-30T06:43:30.219ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129302025-04-30T06:43:30.225ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129312025-04-30T06:43:30.225ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129322025-04-30T06:43:30.225ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129332025-04-30T06:43:30.227ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129342025-04-30T06:43:30.232ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
129352025-04-30T06:43:30.233ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
129362025-04-30T06:43:30.233ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129372025-04-30T06:43:30.234ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 89 fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53106 req_id = 5b5c34c6-d617-46bf-ab59-d1962ff9287a
129382025-04-30T06:43:30.234ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35325,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53106 req_id = 5b5c34c6-d617-46bf-ab59-d1962ff9287a
129392025-04-30T06:43:30.234ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
129402025-04-30T06:43:30.234ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129412025-04-30T06:43:30.235ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53106 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5b5c34c6-d617-46bf-ab59-d1962ff9287a
129422025-04-30T06:43:30.235ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4275, tv_nsec: 34549515 } })
129432025-04-30T06:43:30.235ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
129442025-04-30T06:43:30.237ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b5 a3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51592 req_id = 12ba1b46-0dd2-44bc-b6a9-bed641a91ee7
129452025-04-30T06:43:30.237ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46499,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51592 req_id = 12ba1b46-0dd2-44bc-b6a9-bed641a91ee7
129462025-04-30T06:43:30.237ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129472025-04-30T06:43:30.237ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51592 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 12ba1b46-0dd2-44bc-b6a9-bed641a91ee7
129482025-04-30T06:43:30.290ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
129492025-04-30T06:43:30.291ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129502025-04-30T06:43:30.291ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129512025-04-30T06:43:30.291ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129522025-04-30T06:43:30.291ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
129532025-04-30T06:43:30.292ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129542025-04-30T06:43:30.295ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 49 26 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c I&..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58641 req_id = c9b33b5c-9583-4cf1-a316-56e36d610f4e
129552025-04-30T06:43:30.295ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18726,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58641 req_id = c9b33b5c-9583-4cf1-a316-56e36d610f4e
129562025-04-30T06:43:30.295ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
129572025-04-30T06:43:30.295ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18726, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58641 req_id = c9b33b5c-9583-4cf1-a316-56e36d610f4e
129582025-04-30T06:43:30.296ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
129592025-04-30T06:43:30.296ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
129602025-04-30T06:43:30.296ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
129612025-04-30T06:43:30.298ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129622025-04-30T06:43:30.303ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129632025-04-30T06:43:30.303ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129642025-04-30T06:43:30.303ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
129652025-04-30T06:43:30.304ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129662025-04-30T06:43:30.317ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129672025-04-30T06:43:30.317ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
129682025-04-30T06:43:30.318ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129692025-04-30T06:43:30.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a56e7a3-22e3-4231-82e5-d412f017bf84", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129702025-04-30T06:43:30.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
129712025-04-30T06:43:30.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f2e7d0-8c2c-42c7-977f-c1ab4d694dd5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129722025-04-30T06:43:30.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
129732025-04-30T06:43:30.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a429e45-9718-4866-9c6e-b5d3ac49c673", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129742025-04-30T06:43:30.355ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
129752025-04-30T06:43:30.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6367df-a5f6-435f-8bed-780168c6668d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129762025-04-30T06:43:30.358ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
129772025-04-30T06:43:30.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6f02c24-e9c5-48a8-a369-90068cdda03e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129782025-04-30T06:43:30.361ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
129792025-04-30T06:43:30.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26068a36-2bdc-4a1c-9cdb-9624eb8e0593", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129802025-04-30T06:43:30.365ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
129812025-04-30T06:43:30.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5f5691f-cacb-4019-9582-b2fabd2d64d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129822025-04-30T06:43:30.368ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
129832025-04-30T06:43:30.371ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
129842025-04-30T06:43:30.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b61c66c-c60c-4bcd-869e-6957247b3ec4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129852025-04-30T06:43:30.372ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
129862025-04-30T06:43:30.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3b85a2c-b831-462f-98e5-7d00b17dc97d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129872025-04-30T06:43:30.374ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
129882025-04-30T06:43:30.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eecf819-a9ba-41be-85e6-e99a74329e46", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129892025-04-30T06:43:30.375ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
129902025-04-30T06:43:30.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b33e600a-cbca-4ca0-997d-2c99ca6de9f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129912025-04-30T06:43:30.379ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
129922025-04-30T06:43:30.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3e7930-9467-4031-9ad1-28d953e6b6a8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129932025-04-30T06:43:30.380ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
129942025-04-30T06:43:30.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e166f3c-dda1-4641-878b-9359bfb46eca", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129952025-04-30T06:43:30.383ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
129962025-04-30T06:43:30.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fbfa231-a0a4-4c1d-987e-1f6f8cac42d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129972025-04-30T06:43:30.386ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
129982025-04-30T06:43:30.389ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "446c18d3-1c7b-43ba-99f3-23fdb94a0a9d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
129992025-04-30T06:43:30.390ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
130002025-04-30T06:43:30.393ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93e616ae-b43a-4efe-b48e-281d02382dac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130012025-04-30T06:43:30.393ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
130022025-04-30T06:43:30.396ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f2e32b3-d79b-4af5-8663-b23271d69270", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130032025-04-30T06:43:30.396ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
130042025-04-30T06:43:30.400ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f75fa6dc-ffac-42b9-b4e6-679b6e9699d9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130052025-04-30T06:43:30.400ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
130062025-04-30T06:43:30.403ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f79657c-01fb-4868-8183-0c52410fa5bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130072025-04-30T06:43:30.403ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
130082025-04-30T06:43:30.405ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ce252db-d35d-4a42-8f3f-0c095a7078a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130092025-04-30T06:43:30.405ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
130102025-04-30T06:43:30.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef4d9dab-4f10-480b-a982-a0d2cc941b62", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130112025-04-30T06:43:30.407ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
130122025-04-30T06:43:30.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dcf9a86-4f31-4aa1-b099-a17327bec510", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130132025-04-30T06:43:30.408ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
130142025-04-30T06:43:30.410ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b71dd575-943e-4637-8fdf-d6eda74a1dc1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130152025-04-30T06:43:30.410ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
130162025-04-30T06:43:30.413ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8b12488-4e07-4801-9f16-851be21ff1fa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130172025-04-30T06:43:30.413ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
130182025-04-30T06:43:30.418ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b350fb24-05d6-4a78-b1f2-71e951384af0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130192025-04-30T06:43:30.418ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
130202025-04-30T06:43:30.421ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e89938aa-e758-4aa4-ab6a-a62b40fa2b8a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130212025-04-30T06:43:30.421ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
130222025-04-30T06:43:30.422ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130232025-04-30T06:43:30.422ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130242025-04-30T06:43:30.422ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130252025-04-30T06:43:30.422ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
130262025-04-30T06:43:30.423ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130272025-04-30T06:43:30.425ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f5a6d22-d013-4d67-b5b1-cf7b6a6b065f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130282025-04-30T06:43:30.426ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
130292025-04-30T06:43:30.427ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130302025-04-30T06:43:30.433ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130312025-04-30T06:43:30.433ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130322025-04-30T06:43:30.433ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
130332025-04-30T06:43:30.434ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130342025-04-30T06:43:30.439ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130352025-04-30T06:43:30.439ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
130362025-04-30T06:43:30.443ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130372025-04-30T06:43:30.443ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1c8731-10a8-4949-bea0-9fdedd188c9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130382025-04-30T06:43:30.443ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
130392025-04-30T06:43:30.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdca8853-7a5e-4611-8552-fd0985438665", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130402025-04-30T06:43:30.449ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
130412025-04-30T06:43:30.467ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86146dd6-5b26-490f-b15c-cb68767de86a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130422025-04-30T06:43:30.467ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
130432025-04-30T06:43:30.469ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad3d5a4-e954-4eed-9b69-0e45e61dd41f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130442025-04-30T06:43:30.470ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
130452025-04-30T06:43:30.471ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80847f2f-db0f-4f3b-94c6-5da463b099c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130462025-04-30T06:43:30.471ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
130472025-04-30T06:43:30.474ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1d2647-b4a4-4997-9b46-ef74aa89c3a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130482025-04-30T06:43:30.474ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
130492025-04-30T06:43:30.477ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7edf4768-140c-4501-8956-788c2b5715cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130502025-04-30T06:43:30.477ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
130512025-04-30T06:43:30.479ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c2ffe6a-59b6-4fbc-a2ad-d8de28adf0a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130522025-04-30T06:43:30.480ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
130532025-04-30T06:43:30.487ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
130542025-04-30T06:43:30.488ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efbf3eb5-09c2-4274-986b-1a2a82c98529", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130552025-04-30T06:43:30.488ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
130562025-04-30T06:43:30.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94bc8a6a-5092-47cb-bc46-4d8ea0ca9cf0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130572025-04-30T06:43:30.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
130582025-04-30T06:43:30.496ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e132462-d251-47d1-9050-76066a28223f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130592025-04-30T06:43:30.496ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
130602025-04-30T06:43:30.498ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9114ce22-2442-493c-a3e8-3a57fbd0dc35", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130612025-04-30T06:43:30.498ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
130622025-04-30T06:43:30.502ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5218266f-5e1b-42dc-a344-9ca5677d0fbd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130632025-04-30T06:43:30.502ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
130642025-04-30T06:43:30.508ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c520a4be-d9f2-46b0-b16a-fd729a0f0d7f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130652025-04-30T06:43:30.508ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
130662025-04-30T06:43:30.509ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4459563-716e-4059-9fb2-07f95dbcd1f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130672025-04-30T06:43:30.509ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
130682025-04-30T06:43:30.510ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc3435ca-ad1c-4f5f-9003-dc250cabfd75", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130692025-04-30T06:43:30.510ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
130702025-04-30T06:43:30.512ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82c47d0e-6a40-475f-8853-e73d07e17979", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130712025-04-30T06:43:30.512ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
130722025-04-30T06:43:30.513ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a886a3cb-2615-442d-bd3c-a91923e0f434", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
130732025-04-30T06:43:30.513ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130742025-04-30T06:43:30.513ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130752025-04-30T06:43:30.513ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130762025-04-30T06:43:30.513ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130772025-04-30T06:43:30.515ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130782025-04-30T06:43:30.518ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130792025-04-30T06:43:30.525ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130802025-04-30T06:43:30.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130812025-04-30T06:43:30.525ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130822025-04-30T06:43:30.525ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130832025-04-30T06:43:30.525ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130842025-04-30T06:43:30.528ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130852025-04-30T06:43:30.531ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130862025-04-30T06:43:30.534ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130872025-04-30T06:43:30.535ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130882025-04-30T06:43:30.535ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130892025-04-30T06:43:30.535ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130902025-04-30T06:43:30.540ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130912025-04-30T06:43:30.540ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130922025-04-30T06:43:30.540ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130932025-04-30T06:43:30.540ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
130942025-04-30T06:43:30.541ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130952025-04-30T06:43:30.622ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
130962025-04-30T06:43:30.622ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
130972025-04-30T06:43:30.623ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:60508 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130982025-04-30T06:43:30.629ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130992025-04-30T06:43:30.638ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131002025-04-30T06:43:30.638ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131012025-04-30T06:43:30.638ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
131022025-04-30T06:43:30.640ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131032025-04-30T06:43:30.640ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:60508 req_id = 98b8b089-2eea-4da2-b36e-136680e97821 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131042025-04-30T06:43:30.641ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 1114 local_addr = [::1]:45714 method = GET remote_addr = [::1]:60508 req_id = 98b8b089-2eea-4da2-b36e-136680e97821 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131052025-04-30T06:43:30.642ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
131062025-04-30T06:43:30.644ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b8b089-2eea-4da2-b36e-136680e97821", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:29 GMT"} })
131072025-04-30T06:43:30.647ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131082025-04-30T06:43:30.647ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131092025-04-30T06:43:30.647ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
131102025-04-30T06:43:30.649ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131112025-04-30T06:43:30.649ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131122025-04-30T06:43:30.649ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131132025-04-30T06:43:30.649ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131142025-04-30T06:43:30.676ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131152025-04-30T06:43:30.678ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6c 7a 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 lz..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51124 req_id = 939be83e-55bc-4c0a-9b77-d872609fe6f1
131162025-04-30T06:43:30.679ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27770,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51124 req_id = 939be83e-55bc-4c0a-9b77-d872609fe6f1
131172025-04-30T06:43:30.685ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
131182025-04-30T06:43:30.687ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131192025-04-30T06:43:30.687ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51124 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 939be83e-55bc-4c0a-9b77-d872609fe6f1
131202025-04-30T06:43:30.689ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131212025-04-30T06:43:30.689ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
131222025-04-30T06:43:30.689ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131232025-04-30T06:43:30.690ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4275, tv_nsec: 489063434 } })
131242025-04-30T06:43:30.690ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
131252025-04-30T06:43:30.700ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131262025-04-30T06:43:30.703ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131272025-04-30T06:43:30.704ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131282025-04-30T06:43:30.704ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131292025-04-30T06:43:30.704ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131302025-04-30T06:43:30.710ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131312025-04-30T06:43:30.710ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131322025-04-30T06:43:30.713ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1a 50 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .P..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58235 req_id = a40d3978-bd07-42c3-a74a-f12787e31404
131332025-04-30T06:43:30.713ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6736,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58235 req_id = a40d3978-bd07-42c3-a74a-f12787e31404
131342025-04-30T06:43:30.715ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131352025-04-30T06:43:30.716ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58235 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a40d3978-bd07-42c3-a74a-f12787e31404
131362025-04-30T06:43:30.733ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
131372025-04-30T06:43:30.804ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
131382025-04-30T06:43:30.804ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
131392025-04-30T06:43:30.804ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
131402025-04-30T06:43:30.804ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
131412025-04-30T06:43:30.805ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
131422025-04-30T06:43:30.806ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131432025-04-30T06:43:30.807ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131442025-04-30T06:43:30.807ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131452025-04-30T06:43:30.807ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131462025-04-30T06:43:30.808ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
131472025-04-30T06:43:30.808ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f0 34 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .4..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63108 req_id = 7aa1e695-1ab4-493f-9328-00219d2abd3d
131482025-04-30T06:43:30.808ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61492,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63108 req_id = 7aa1e695-1ab4-493f-9328-00219d2abd3d
131492025-04-30T06:43:30.809ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
131502025-04-30T06:43:30.809ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131512025-04-30T06:43:30.809ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63108 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7aa1e695-1ab4-493f-9328-00219d2abd3d
131522025-04-30T06:43:30.810ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4275, tv_nsec: 609337242 } })
131532025-04-30T06:43:30.810ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
131542025-04-30T06:43:30.810ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 90 a3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51552 req_id = e9107582-b11e-4285-9b98-c74aeef3c543
131552025-04-30T06:43:30.811ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37027,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51552 req_id = e9107582-b11e-4285-9b98-c74aeef3c543
131562025-04-30T06:43:30.811ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131572025-04-30T06:43:30.811ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51552 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e9107582-b11e-4285-9b98-c74aeef3c543
131582025-04-30T06:43:30.813ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
131592025-04-30T06:43:30.887ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
131602025-04-30T06:43:30.888ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
131612025-04-30T06:43:30.889ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
131622025-04-30T06:43:30.889ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
131632025-04-30T06:43:30.891ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
131642025-04-30T06:43:30.930ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
131652025-04-30T06:43:30.930ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
131662025-04-30T06:43:30.930ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 715.499686ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
131672025-04-30T06:43:30.931ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
131682025-04-30T06:43:30.931ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131692025-04-30T06:43:30.931ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131702025-04-30T06:43:30.931ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131712025-04-30T06:43:30.931ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131722025-04-30T06:43:30.931ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131732025-04-30T06:43:30.936ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131742025-04-30T06:43:30.946ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131752025-04-30T06:43:30.946ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131762025-04-30T06:43:30.946ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131772025-04-30T06:43:30.947ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131782025-04-30T06:43:30.953ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
131792025-04-30T06:43:30.965ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
131802025-04-30T06:43:30.965ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
131812025-04-30T06:43:30.966ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 61 1c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 a............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63469 req_id = c742cd7b-716e-4c3a-ad9e-f4eb29ea4f3d
131822025-04-30T06:43:30.966ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24860,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63469 req_id = c742cd7b-716e-4c3a-ad9e-f4eb29ea4f3d
131832025-04-30T06:43:30.966ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
131842025-04-30T06:43:30.967ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131852025-04-30T06:43:30.967ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63469 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c742cd7b-716e-4c3a-ad9e-f4eb29ea4f3d
131862025-04-30T06:43:30.967ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4275, tv_nsec: 766802100 } })
131872025-04-30T06:43:30.967ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
131882025-04-30T06:43:30.968ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4b f1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e K...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49827 req_id = 67bb279d-9408-4234-8d5d-879656d51c3a
131892025-04-30T06:43:30.968ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19441,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49827 req_id = 67bb279d-9408-4234-8d5d-879656d51c3a
131902025-04-30T06:43:30.968ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131912025-04-30T06:43:30.968ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49827 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 67bb279d-9408-4234-8d5d-879656d51c3a
131922025-04-30T06:43:31.037ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
131932025-04-30T06:43:31.039ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131942025-04-30T06:43:31.039ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131952025-04-30T06:43:31.039ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131962025-04-30T06:43:31.039ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
131972025-04-30T06:43:31.041ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131982025-04-30T06:43:31.041ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
131992025-04-30T06:43:31.041ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132002025-04-30T06:43:31.042ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
132012025-04-30T06:43:31.043ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
132022025-04-30T06:43:31.043ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b9 c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61955 req_id = 2eac647d-f6fc-4197-9d85-9ed8dd58d5b1
132032025-04-30T06:43:31.043ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47552,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61955 req_id = 2eac647d-f6fc-4197-9d85-9ed8dd58d5b1
132042025-04-30T06:43:31.045ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
132052025-04-30T06:43:31.045ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47552, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61955 req_id = 2eac647d-f6fc-4197-9d85-9ed8dd58d5b1
132062025-04-30T06:43:31.045ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132072025-04-30T06:43:31.052ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132082025-04-30T06:43:31.052ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132092025-04-30T06:43:31.053ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
132102025-04-30T06:43:31.053ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132112025-04-30T06:43:31.053ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
132122025-04-30T06:43:31.053ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
132132025-04-30T06:43:31.053ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
132142025-04-30T06:43:31.058ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132152025-04-30T06:43:31.058ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
132162025-04-30T06:43:31.059ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132172025-04-30T06:43:31.059ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
132182025-04-30T06:43:31.059ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
132192025-04-30T06:43:31.059ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e6bb03-f308-4d60-968c-db93a8a915ca", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132202025-04-30T06:43:31.060ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
132212025-04-30T06:43:31.064ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe98a614-07fa-4b99-a2fd-a3b4aea47ed6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132222025-04-30T06:43:31.065ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
132232025-04-30T06:43:31.068ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
132242025-04-30T06:43:31.068ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132252025-04-30T06:43:31.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d955437-9e17-427e-b52a-270b444e5b67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132262025-04-30T06:43:31.075ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
132272025-04-30T06:43:31.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a0f8819-ed19-4ef3-a071-1955d995a8a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132282025-04-30T06:43:31.080ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
132292025-04-30T06:43:31.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "727f29b2-16ad-4585-afcf-f9d13306cea0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132302025-04-30T06:43:31.082ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
132312025-04-30T06:43:31.084ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ee4547f-d16a-4f9a-bc9a-35af988bf4bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132322025-04-30T06:43:31.085ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
132332025-04-30T06:43:31.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fea95391-4b92-4183-9a1f-1003c03a1a8e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132342025-04-30T06:43:31.087ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
132352025-04-30T06:43:31.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b267bb8d-b9c0-48de-a87b-27a78b1d9880", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132362025-04-30T06:43:31.089ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
132372025-04-30T06:43:31.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abba7437-d38a-4fef-a1fd-c58e5d6572bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132382025-04-30T06:43:31.092ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
132392025-04-30T06:43:31.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21d5735-8295-40f2-8f5f-b151bc512545", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132402025-04-30T06:43:31.093ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
132412025-04-30T06:43:31.094ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f36dc123-5012-428b-aff0-f735c5a3bacf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132422025-04-30T06:43:31.095ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
132432025-04-30T06:43:31.095ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "656fd57a-5083-4dd6-a2f7-640460da48ed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132442025-04-30T06:43:31.096ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
132452025-04-30T06:43:31.098ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29ba2031-bdc8-48a2-8c3f-456c5d28903d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132462025-04-30T06:43:31.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
132472025-04-30T06:43:31.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af1fd93d-aab8-44d0-941a-af141405e282", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132482025-04-30T06:43:31.101ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
132492025-04-30T06:43:31.103ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b32018b-68cf-4e92-bea0-7766bdf2da3b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132502025-04-30T06:43:31.103ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
132512025-04-30T06:43:31.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1935acc-5be8-4da3-be7e-20a9d26a59aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132522025-04-30T06:43:31.106ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
132532025-04-30T06:43:31.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dacedb00-763c-43fb-8071-783816f2972f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132542025-04-30T06:43:31.108ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
132552025-04-30T06:43:31.110ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1e5596-9b98-4b67-b67a-ef1d42f0f684", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132562025-04-30T06:43:31.111ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
132572025-04-30T06:43:31.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f92e45d-4e31-4794-bcdf-df2fe64674db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132582025-04-30T06:43:31.113ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
132592025-04-30T06:43:31.115ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21010560-2e30-4757-8277-8f1edb628a7e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132602025-04-30T06:43:31.115ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
132612025-04-30T06:43:31.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e9ea6a-986f-47a8-b533-00ea87a44c4b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132622025-04-30T06:43:31.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
132632025-04-30T06:43:31.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12b49455-59b1-4684-83a9-6ebc29c70471", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132642025-04-30T06:43:31.122ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
132652025-04-30T06:43:31.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ae3b70-4d84-4235-ac1c-2dcc75c0f423", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132662025-04-30T06:43:31.124ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
132672025-04-30T06:43:31.126ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94f2e627-4ba9-4b89-9f7b-6186a6d95ee4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132682025-04-30T06:43:31.127ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
132692025-04-30T06:43:31.131ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae01ad5-807e-4a25-8663-9c982ef34b69", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132702025-04-30T06:43:31.131ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
132712025-04-30T06:43:31.135ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c8a48f5-7af2-4006-ac90-3e31a776adb3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132722025-04-30T06:43:31.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
132732025-04-30T06:43:31.136ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
132742025-04-30T06:43:31.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e7f2e6-7e55-4821-9f31-bff365bcf637", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132752025-04-30T06:43:31.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
132762025-04-30T06:43:31.142ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
132772025-04-30T06:43:31.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78109e84-f3e2-4a31-9c60-89f176d11747", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132782025-04-30T06:43:31.143ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
132792025-04-30T06:43:31.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed79416-ab1c-4f65-a78f-4d2e6671e017", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132802025-04-30T06:43:31.145ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
132812025-04-30T06:43:31.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d83d2fe-2886-40ec-97e5-05d12ab1f6c3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132822025-04-30T06:43:31.148ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
132832025-04-30T06:43:31.149ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "811f25c8-c622-4edf-a36a-e97cdce8e858", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132842025-04-30T06:43:31.150ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
132852025-04-30T06:43:31.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7641bfe-3a3f-4885-8ffb-e04ffb83b71b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132862025-04-30T06:43:31.151ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
132872025-04-30T06:43:31.152ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e5a28ab-628b-4620-84be-05f32731e6be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132882025-04-30T06:43:31.153ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
132892025-04-30T06:43:31.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6237255a-17d3-47de-955b-4618dbeff528", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132902025-04-30T06:43:31.154ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
132912025-04-30T06:43:31.155ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b3f38ec-95fc-48f9-9378-7199f580b3a3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132922025-04-30T06:43:31.155ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
132932025-04-30T06:43:31.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d49dfd9-1d76-4160-90fd-7cfdb707e71c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132942025-04-30T06:43:31.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
132952025-04-30T06:43:31.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "832e57ff-2c99-4b2e-9f47-e17b6ae41d7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132962025-04-30T06:43:31.159ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
132972025-04-30T06:43:31.160ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95677b7f-d67f-4da7-af9c-7ee8d3c0b7fe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
132982025-04-30T06:43:31.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
132992025-04-30T06:43:31.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "488afe80-29cf-497b-bd50-281c23a20863", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133002025-04-30T06:43:31.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
133012025-04-30T06:43:31.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e390d8d3-f247-4224-9c3a-7db54dbf26aa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133022025-04-30T06:43:31.165ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
133032025-04-30T06:43:31.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e32d2326-999a-4201-817f-af54b0560238", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133042025-04-30T06:43:31.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
133052025-04-30T06:43:31.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40486ddd-31b5-4f22-bb7b-31fd4ae20008", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133062025-04-30T06:43:31.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
133072025-04-30T06:43:31.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a046730b-3b80-4012-a15b-287ee0a876c6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133082025-04-30T06:43:31.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
133092025-04-30T06:43:31.170ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17f159a-531e-4df5-a89a-2cdd42d6c8c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133102025-04-30T06:43:31.171ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
133112025-04-30T06:43:31.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6df6f80c-7ea0-431a-bace-8f6e4d955392", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133122025-04-30T06:43:31.172ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133132025-04-30T06:43:31.172ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133142025-04-30T06:43:31.172ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133152025-04-30T06:43:31.172ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133162025-04-30T06:43:31.173ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133172025-04-30T06:43:31.177ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133182025-04-30T06:43:31.184ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133192025-04-30T06:43:31.184ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133202025-04-30T06:43:31.184ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133212025-04-30T06:43:31.184ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133222025-04-30T06:43:31.184ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133232025-04-30T06:43:31.185ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133242025-04-30T06:43:31.194ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133252025-04-30T06:43:31.194ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133262025-04-30T06:43:31.194ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133272025-04-30T06:43:31.194ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
133282025-04-30T06:43:31.195ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133292025-04-30T06:43:31.197ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133302025-04-30T06:43:31.201ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133312025-04-30T06:43:31.201ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133322025-04-30T06:43:31.201ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133332025-04-30T06:43:31.201ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133342025-04-30T06:43:31.208ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133352025-04-30T06:43:31.212ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133362025-04-30T06:43:31.213ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133372025-04-30T06:43:31.213ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
133382025-04-30T06:43:31.213ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133392025-04-30T06:43:31.261ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
133402025-04-30T06:43:31.261ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
133412025-04-30T06:43:31.262ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:52426 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
133422025-04-30T06:43:31.264ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:52426 req_id = 7a1af5ee-fc8f-446e-afb7-e1843903824a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133432025-04-30T06:43:31.264ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 647 local_addr = [::1]:45714 method = GET remote_addr = [::1]:52426 req_id = 7a1af5ee-fc8f-446e-afb7-e1843903824a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133442025-04-30T06:43:31.264ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133452025-04-30T06:43:31.265ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
133462025-04-30T06:43:31.266ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133472025-04-30T06:43:31.267ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a1af5ee-fc8f-446e-afb7-e1843903824a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
133482025-04-30T06:43:31.270ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133492025-04-30T06:43:31.270ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133502025-04-30T06:43:31.270ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
133512025-04-30T06:43:31.277ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133522025-04-30T06:43:31.277ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133532025-04-30T06:43:31.277ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133542025-04-30T06:43:31.277ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133552025-04-30T06:43:31.278ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133562025-04-30T06:43:31.297ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133572025-04-30T06:43:31.302ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133582025-04-30T06:43:31.302ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133592025-04-30T06:43:31.302ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133602025-04-30T06:43:31.302ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133612025-04-30T06:43:31.383ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
133622025-04-30T06:43:31.388ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
133632025-04-30T06:43:31.389ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
133642025-04-30T06:43:31.389ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 458.372698ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
133652025-04-30T06:43:31.389ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
133662025-04-30T06:43:31.389ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133672025-04-30T06:43:31.389ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133682025-04-30T06:43:31.389ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133692025-04-30T06:43:31.389ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133702025-04-30T06:43:31.390ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133712025-04-30T06:43:31.395ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133722025-04-30T06:43:31.399ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133732025-04-30T06:43:31.399ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133742025-04-30T06:43:31.399ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133752025-04-30T06:43:31.401ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133762025-04-30T06:43:31.405ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
133772025-04-30T06:43:31.405ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
133782025-04-30T06:43:31.406ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c8 fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64984 req_id = af8f3a75-bac8-433e-b677-2ba313b8325e
133792025-04-30T06:43:31.406ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51453,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64984 req_id = af8f3a75-bac8-433e-b677-2ba313b8325e
133802025-04-30T06:43:31.407ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
133812025-04-30T06:43:31.407ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133822025-04-30T06:43:31.407ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64984 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = af8f3a75-bac8-433e-b677-2ba313b8325e
133832025-04-30T06:43:31.407ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4276, tv_nsec: 206599498 } })
133842025-04-30T06:43:31.407ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
133852025-04-30T06:43:31.408ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 33 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .3..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49498 req_id = 7714a1b6-1065-436b-9c39-b7a45283447f
133862025-04-30T06:43:31.408ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49498 req_id = 7714a1b6-1065-436b-9c39-b7a45283447f
133872025-04-30T06:43:31.408ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133882025-04-30T06:43:31.408ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49498 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7714a1b6-1065-436b-9c39-b7a45283447f
133892025-04-30T06:43:31.469ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
133902025-04-30T06:43:31.470ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133912025-04-30T06:43:31.470ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133922025-04-30T06:43:31.470ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133932025-04-30T06:43:31.470ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
133942025-04-30T06:43:31.471ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133952025-04-30T06:43:31.471ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e4 fa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65311 req_id = 18682d63-1907-466b-980c-564d35da58dd
133962025-04-30T06:43:31.471ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58618,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65311 req_id = 18682d63-1907-466b-980c-564d35da58dd
133972025-04-30T06:43:31.471ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
133982025-04-30T06:43:31.472ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58618, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65311 req_id = 18682d63-1907-466b-980c-564d35da58dd
133992025-04-30T06:43:31.472ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
134002025-04-30T06:43:31.472ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
134012025-04-30T06:43:31.472ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
134022025-04-30T06:43:31.474ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c94461-ae3a-48cc-a2c0-00eff7dcac78", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134032025-04-30T06:43:31.474ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
134042025-04-30T06:43:31.475ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134052025-04-30T06:43:31.479ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134062025-04-30T06:43:31.479ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134072025-04-30T06:43:31.479ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
134082025-04-30T06:43:31.480ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134092025-04-30T06:43:31.482ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48a4e7fd-82f2-45d0-a433-f3afaf2b6b87", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134102025-04-30T06:43:31.482ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
134112025-04-30T06:43:31.482ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134122025-04-30T06:43:31.482ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
134132025-04-30T06:43:31.483ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134142025-04-30T06:43:31.485ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "655e9bf4-0ed1-47a5-966b-b97d88522b1f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134152025-04-30T06:43:31.486ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
134162025-04-30T06:43:31.489ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed16635-e47c-419c-956f-0a231175c96e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134172025-04-30T06:43:31.489ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
134182025-04-30T06:43:31.491ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff1d7ff-0336-446a-8e8d-7f5f6907c031", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134192025-04-30T06:43:31.493ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
134202025-04-30T06:43:31.495ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "982890bf-aa1e-4d69-bcaa-07a24c4a55ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134212025-04-30T06:43:31.495ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
134222025-04-30T06:43:31.501ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c3902e0-8c6f-42bc-9954-72123a2dc09a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134232025-04-30T06:43:31.501ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
134242025-04-30T06:43:31.504ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41de5576-3f93-4cbb-aebb-c0b5b8811341", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134252025-04-30T06:43:31.504ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
134262025-04-30T06:43:31.506ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61e2bfa3-7ce7-4c12-a58c-d0ce2182a7fe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134272025-04-30T06:43:31.506ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
134282025-04-30T06:43:31.508ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06608036-283f-419a-b197-15d13e0b3327", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134292025-04-30T06:43:31.508ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
134302025-04-30T06:43:31.511ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37851932-8b79-42f5-9f72-e172420ac06a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134312025-04-30T06:43:31.511ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
134322025-04-30T06:43:31.512ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7c2dbc-6b34-49e5-92ee-6254c4588eb5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134332025-04-30T06:43:31.512ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
134342025-04-30T06:43:31.518ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
134352025-04-30T06:43:31.521ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81dc4882-126d-4fae-aeda-415de81df78c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134362025-04-30T06:43:31.521ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
134372025-04-30T06:43:31.528ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "527d1e14-573c-4ed1-9aff-bb6dce0a4498", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134382025-04-30T06:43:31.529ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
134392025-04-30T06:43:31.532ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e822bb9c-da60-4691-a38d-af9f9ecb184d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134402025-04-30T06:43:31.532ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
134412025-04-30T06:43:31.538ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12903610-5b77-434a-8151-f0f442121d33", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134422025-04-30T06:43:31.538ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
134432025-04-30T06:43:31.547ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31296c55-95f3-450b-b562-558011e9cd70", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134442025-04-30T06:43:31.547ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
134452025-04-30T06:43:31.554ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbadb4f2-93a1-453d-80c0-77568dd49af7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134462025-04-30T06:43:31.554ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
134472025-04-30T06:43:31.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1c18ac-2916-45d1-a88e-c83d7a29fe83", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134482025-04-30T06:43:31.558ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
134492025-04-30T06:43:31.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1130e966-130b-46a9-8c59-a736586866cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134502025-04-30T06:43:31.567ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
134512025-04-30T06:43:31.569ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a25a7ba7-9312-42a5-a162-e42f8d829705", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134522025-04-30T06:43:31.569ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
134532025-04-30T06:43:31.570ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134542025-04-30T06:43:31.570ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134552025-04-30T06:43:31.570ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134562025-04-30T06:43:31.570ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
134572025-04-30T06:43:31.570ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134582025-04-30T06:43:31.574ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb201953-279d-48ff-9827-3ac0aea49b42", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134592025-04-30T06:43:31.574ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
134602025-04-30T06:43:31.575ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134612025-04-30T06:43:31.629ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134622025-04-30T06:43:31.629ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134632025-04-30T06:43:31.629ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
134642025-04-30T06:43:31.631ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134652025-04-30T06:43:31.637ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
134662025-04-30T06:43:31.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57db0901-8ca1-4d3a-b20f-203cf680a03c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134672025-04-30T06:43:31.638ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
134682025-04-30T06:43:31.639ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134692025-04-30T06:43:31.640ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 71a7bfe5-64f3-4340-b8bc-5d498f896e1a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134702025-04-30T06:43:31.641ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 71a7bfe5-64f3-4340-b8bc-5d498f896e1a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134712025-04-30T06:43:31.641ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 71a7bfe5-64f3-4340-b8bc-5d498f896e1a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134722025-04-30T06:43:31.642ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 71a7bfe5-64f3-4340-b8bc-5d498f896e1a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134732025-04-30T06:43:31.645ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134742025-04-30T06:43:31.646ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
134752025-04-30T06:43:31.646ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134762025-04-30T06:43:31.648ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 8485 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 71a7bfe5-64f3-4340-b8bc-5d498f896e1a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
134772025-04-30T06:43:31.651ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a7bfe5-64f3-4340-b8bc-5d498f896e1a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134782025-04-30T06:43:31.652ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134792025-04-30T06:43:31.653ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d2cd28f3-053d-471d-ae26-0153840551d4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134802025-04-30T06:43:31.653ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d2cd28f3-053d-471d-ae26-0153840551d4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134812025-04-30T06:43:31.653ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d2cd28f3-053d-471d-ae26-0153840551d4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134822025-04-30T06:43:31.656ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d2cd28f3-053d-471d-ae26-0153840551d4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134832025-04-30T06:43:31.657ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94bbd254-aecd-46e9-a651-c18d4c2e9be0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134842025-04-30T06:43:31.658ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
134852025-04-30T06:43:31.659ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 5722 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = d2cd28f3-053d-471d-ae26-0153840551d4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
134862025-04-30T06:43:31.660ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2cd28f3-053d-471d-ae26-0153840551d4", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134872025-04-30T06:43:31.660ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
134882025-04-30T06:43:31.660ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
134892025-04-30T06:43:31.660ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
134902025-04-30T06:43:31.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "662b5fca-f30e-477e-ae50-bf3d54cfaf38", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134912025-04-30T06:43:31.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
134922025-04-30T06:43:31.678ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c3391d-bbcf-4c83-b3d8-0a0bba23f034", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:30 GMT"} })
134932025-04-30T06:43:31.678ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
134942025-04-30T06:43:31.681ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cde2d81-127c-4592-8342-583a6c022cb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
134952025-04-30T06:43:31.681ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
134962025-04-30T06:43:31.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2e5b1cf-7657-4bf3-88b6-1202e75ae361", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
134972025-04-30T06:43:31.684ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
134982025-04-30T06:43:31.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34161ba0-307f-4573-834e-2a98333396a1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
134992025-04-30T06:43:31.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
135002025-04-30T06:43:31.694ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
135012025-04-30T06:43:31.713ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135022025-04-30T06:43:31.713ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135032025-04-30T06:43:31.716ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135042025-04-30T06:43:31.720ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135052025-04-30T06:43:31.721ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135062025-04-30T06:43:31.721ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135072025-04-30T06:43:31.722ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0393e16d-f1b9-4ae9-a072-062972703a59", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135082025-04-30T06:43:31.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
135092025-04-30T06:43:31.725ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "024e7d46-5462-4c5a-8d81-7e464365f2b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135102025-04-30T06:43:31.727ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
135112025-04-30T06:43:31.730ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a97e29ec-4df2-4a65-a9ef-6f1b1131a94c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135122025-04-30T06:43:31.730ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
135132025-04-30T06:43:31.732ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e920ef-f67a-4b0e-8037-1511aaf8179d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135142025-04-30T06:43:31.732ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
135152025-04-30T06:43:31.734ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6045d5be-efe4-4127-8d9b-051a9ea714bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135162025-04-30T06:43:31.735ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
135172025-04-30T06:43:31.737ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68c69f91-f01b-4286-9a7c-2a899a2d5d2c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135182025-04-30T06:43:31.738ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
135192025-04-30T06:43:31.742ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a11925e-ee71-4508-a213-3bf8b610b2f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135202025-04-30T06:43:31.742ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
135212025-04-30T06:43:31.745ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135222025-04-30T06:43:31.745ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135232025-04-30T06:43:31.745ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135242025-04-30T06:43:31.746ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
135252025-04-30T06:43:31.747ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135262025-04-30T06:43:31.751ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135272025-04-30T06:43:31.759ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135282025-04-30T06:43:31.759ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135292025-04-30T06:43:31.759ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
135302025-04-30T06:43:31.761ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135312025-04-30T06:43:31.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be26bb3-a55b-4f55-9854-0458f1d8a97a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135322025-04-30T06:43:31.764ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
135332025-04-30T06:43:31.765ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135342025-04-30T06:43:31.765ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
135352025-04-30T06:43:31.766ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135362025-04-30T06:43:31.782ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5798403f-f6c4-45eb-a869-3c40cbf4cc6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135372025-04-30T06:43:31.783ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
135382025-04-30T06:43:31.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8abe8e-25e5-44b0-a2c9-ad3f0f5e1a0f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135392025-04-30T06:43:31.787ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
135402025-04-30T06:43:31.793ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "baa9031f-10ea-461a-bdb5-ba15b3805f72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135412025-04-30T06:43:31.795ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
135422025-04-30T06:43:31.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f29dade-ac68-4cca-b099-6cec56130dc3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135432025-04-30T06:43:31.809ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
135442025-04-30T06:43:31.811ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afa6820a-dd7b-44b0-83c3-6344eb7748e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135452025-04-30T06:43:31.813ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
135462025-04-30T06:43:31.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "230d9531-ceca-48db-a662-7275d19bf012", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135472025-04-30T06:43:31.817ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
135482025-04-30T06:43:31.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4ed810-07e0-4c38-a02f-8612f3caf46b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135492025-04-30T06:43:31.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
135502025-04-30T06:43:31.824ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
135512025-04-30T06:43:31.827ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a18f33fe-447f-4cc3-ab6c-0a38498491aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135522025-04-30T06:43:31.827ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135532025-04-30T06:43:31.827ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135542025-04-30T06:43:31.827ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135552025-04-30T06:43:31.827ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135562025-04-30T06:43:31.828ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135572025-04-30T06:43:31.835ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135582025-04-30T06:43:31.841ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135592025-04-30T06:43:31.841ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135602025-04-30T06:43:31.841ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135612025-04-30T06:43:31.841ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135622025-04-30T06:43:31.841ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135632025-04-30T06:43:31.842ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135642025-04-30T06:43:31.848ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135652025-04-30T06:43:31.855ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135662025-04-30T06:43:31.857ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135672025-04-30T06:43:31.858ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135682025-04-30T06:43:31.859ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135692025-04-30T06:43:31.925ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
135702025-04-30T06:43:31.925ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
135712025-04-30T06:43:31.927ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135722025-04-30T06:43:31.927ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135732025-04-30T06:43:31.927ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135742025-04-30T06:43:31.927ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
135752025-04-30T06:43:31.928ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135762025-04-30T06:43:31.928ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:63823 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135772025-04-30T06:43:31.932ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:63823 req_id = 14d5baf3-de68-4d8a-b12f-bd5a502cc0bb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135782025-04-30T06:43:31.932ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 658 local_addr = [::1]:45714 method = GET remote_addr = [::1]:63823 req_id = 14d5baf3-de68-4d8a-b12f-bd5a502cc0bb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135792025-04-30T06:43:31.933ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d5baf3-de68-4d8a-b12f-bd5a502cc0bb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
135802025-04-30T06:43:31.936ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135812025-04-30T06:43:31.936ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135822025-04-30T06:43:31.936ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
135832025-04-30T06:43:31.939ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135842025-04-30T06:43:31.939ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135852025-04-30T06:43:31.939ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135862025-04-30T06:43:31.939ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135872025-04-30T06:43:31.940ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135882025-04-30T06:43:31.941ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135892025-04-30T06:43:31.946ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135902025-04-30T06:43:31.946ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135912025-04-30T06:43:31.946ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
135922025-04-30T06:43:31.946ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135932025-04-30T06:43:31.948ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135942025-04-30T06:43:31.953ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135952025-04-30T06:43:31.953ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135962025-04-30T06:43:31.953ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135972025-04-30T06:43:31.954ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135982025-04-30T06:43:31.963ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135992025-04-30T06:43:31.963ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
136002025-04-30T06:43:31.964ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136012025-04-30T06:43:31.975ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136022025-04-30T06:43:31.975ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136032025-04-30T06:43:31.975ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
136042025-04-30T06:43:31.977ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
136052025-04-30T06:43:31.983ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
136062025-04-30T06:43:31.983ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
136072025-04-30T06:43:31.987ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
136082025-04-30T06:43:31.987ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136092025-04-30T06:43:32.004ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
136102025-04-30T06:43:32.004ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
136112025-04-30T06:43:32.005ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
136122025-04-30T06:43:32.005ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 616.038837ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
136132025-04-30T06:43:32.005ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
136142025-04-30T06:43:32.005ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136152025-04-30T06:43:32.005ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136162025-04-30T06:43:32.005ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136172025-04-30T06:43:32.005ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136182025-04-30T06:43:32.015ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136192025-04-30T06:43:32.017ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136202025-04-30T06:43:32.024ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136212025-04-30T06:43:32.024ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136222025-04-30T06:43:32.024ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136232025-04-30T06:43:32.025ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136242025-04-30T06:43:32.029ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
136252025-04-30T06:43:32.029ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
136262025-04-30T06:43:32.031ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 86 aa 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51867 req_id = 2cbe96db-7448-42b5-a44c-2b19f468f99e
136272025-04-30T06:43:32.031ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34474,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51867 req_id = 2cbe96db-7448-42b5-a44c-2b19f468f99e
136282025-04-30T06:43:32.031ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
136292025-04-30T06:43:32.031ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136302025-04-30T06:43:32.031ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51867 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2cbe96db-7448-42b5-a44c-2b19f468f99e
136312025-04-30T06:43:32.032ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4276, tv_nsec: 831285574 } })
136322025-04-30T06:43:32.032ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
136332025-04-30T06:43:32.032ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b1 59 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61730 req_id = fe1ceb59-6e5c-42b7-b241-1ba98dd6f45d
136342025-04-30T06:43:32.032ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45401,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61730 req_id = fe1ceb59-6e5c-42b7-b241-1ba98dd6f45d
136352025-04-30T06:43:32.033ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136362025-04-30T06:43:32.033ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61730 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fe1ceb59-6e5c-42b7-b241-1ba98dd6f45d
136372025-04-30T06:43:32.100ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
136382025-04-30T06:43:32.100ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136392025-04-30T06:43:32.100ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136402025-04-30T06:43:32.100ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136412025-04-30T06:43:32.100ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
136422025-04-30T06:43:32.102ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136432025-04-30T06:43:32.103ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
136442025-04-30T06:43:32.103ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 65 3a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c e:..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60645 req_id = 635cd650-faf0-43cb-88de-2288c234682c
136452025-04-30T06:43:32.103ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25914,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60645 req_id = 635cd650-faf0-43cb-88de-2288c234682c
136462025-04-30T06:43:32.103ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
136472025-04-30T06:43:32.103ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60645 req_id = 635cd650-faf0-43cb-88de-2288c234682c
136482025-04-30T06:43:32.104ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
136492025-04-30T06:43:32.104ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
136502025-04-30T06:43:32.104ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
136512025-04-30T06:43:32.108ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136522025-04-30T06:43:32.116ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136532025-04-30T06:43:32.116ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136542025-04-30T06:43:32.116ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
136552025-04-30T06:43:32.116ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136562025-04-30T06:43:32.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a2f4fe-bcda-463a-ba20-95c0ea7cace2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136572025-04-30T06:43:32.117ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
136582025-04-30T06:43:32.121ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136592025-04-30T06:43:32.121ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
136602025-04-30T06:43:32.121ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136612025-04-30T06:43:32.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd605455-c0b9-4fc2-a5f2-0e5cd6ed2660", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136622025-04-30T06:43:32.123ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
136632025-04-30T06:43:32.125ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:40819 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136642025-04-30T06:43:32.125ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:40819 req_id = 8e8e5e27-3f6a-44c5-86e2-f4934ec6d0f2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
136652025-04-30T06:43:32.126ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 389 local_addr = [::1]:61186 method = GET remote_addr = [::1]:40819 req_id = 8e8e5e27-3f6a-44c5-86e2-f4934ec6d0f2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
136662025-04-30T06:43:32.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1a03c74-7b56-4871-b19c-3f0730aa591c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136672025-04-30T06:43:32.130ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
136682025-04-30T06:43:32.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "396c213b-0ab8-4748-8ef4-70aac7b9ef6b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136692025-04-30T06:43:32.136ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
136702025-04-30T06:43:32.141ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d64356f5-8878-484a-b64e-94a5826662c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136712025-04-30T06:43:32.142ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
136722025-04-30T06:43:32.146ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed1787b-fa1b-4abe-95db-64c902524d74", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136732025-04-30T06:43:32.147ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
136742025-04-30T06:43:32.156ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
136752025-04-30T06:43:32.156ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1227b858-4b78-4320-878e-6a145c31d15a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136762025-04-30T06:43:32.157ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
136772025-04-30T06:43:32.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f31a64-da4d-4959-9288-3d19f2151a67", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136782025-04-30T06:43:32.161ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
136792025-04-30T06:43:32.162ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98543aa5-03c6-422b-b79b-9d6afc343975", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136802025-04-30T06:43:32.163ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
136812025-04-30T06:43:32.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4ef0e26-d56b-4d08-b9a3-3eb013f398ce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136822025-04-30T06:43:32.164ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
136832025-04-30T06:43:32.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d96bde1-98e1-4ee0-bb83-39ab861223d8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136842025-04-30T06:43:32.166ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
136852025-04-30T06:43:32.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5f6525-115d-45f5-9244-03fd5fe75052", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136862025-04-30T06:43:32.167ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
136872025-04-30T06:43:32.168ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a973b2-4fdb-4481-9ae6-5e6154c7436c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136882025-04-30T06:43:32.169ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
136892025-04-30T06:43:32.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41ae3da1-966a-4791-9137-e3ea5f001325", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136902025-04-30T06:43:32.172ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
136912025-04-30T06:43:32.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9973dde3-762b-4e6f-9bb5-0332fb66b029", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136922025-04-30T06:43:32.174ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
136932025-04-30T06:43:32.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da59084-6f50-4a02-bee4-71fa2a9e1ebf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
136942025-04-30T06:43:32.176ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
136952025-04-30T06:43:32.177ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
136962025-04-30T06:43:32.178ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 23 27 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 #'..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56876 req_id = df643c55-2b28-447b-bb9e-07af95df2fcd
136972025-04-30T06:43:32.178ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8999,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56876 req_id = df643c55-2b28-447b-bb9e-07af95df2fcd
136982025-04-30T06:43:32.179ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
136992025-04-30T06:43:32.179ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137002025-04-30T06:43:32.179ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56876 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = df643c55-2b28-447b-bb9e-07af95df2fcd
137012025-04-30T06:43:32.179ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4276, tv_nsec: 978832702 } })
137022025-04-30T06:43:32.179ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
137032025-04-30T06:43:32.180ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 21 de 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e !...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51311 req_id = dbbd2fc1-da74-48df-ac1b-381eeca63f75
137042025-04-30T06:43:32.180ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8670,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51311 req_id = dbbd2fc1-da74-48df-ac1b-381eeca63f75
137052025-04-30T06:43:32.180ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137062025-04-30T06:43:32.180ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51311 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dbbd2fc1-da74-48df-ac1b-381eeca63f75
137072025-04-30T06:43:32.181ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
137082025-04-30T06:43:32.242ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
137092025-04-30T06:43:32.242ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
137102025-04-30T06:43:32.243ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
137112025-04-30T06:43:32.243ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
137122025-04-30T06:43:32.245ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
137132025-04-30T06:43:32.246ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137142025-04-30T06:43:32.246ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137152025-04-30T06:43:32.246ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137162025-04-30T06:43:32.246ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
137172025-04-30T06:43:32.247ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137182025-04-30T06:43:32.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a7daae-1ed1-4c75-bd7e-e463b803d41f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137192025-04-30T06:43:32.249ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
137202025-04-30T06:43:32.250ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137212025-04-30T06:43:32.257ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137222025-04-30T06:43:32.257ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137232025-04-30T06:43:32.257ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
137242025-04-30T06:43:32.258ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137252025-04-30T06:43:32.264ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137262025-04-30T06:43:32.264ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
137272025-04-30T06:43:32.265ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137282025-04-30T06:43:32.286ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ba50af8-ea8c-4cc4-b436-bab5233cf123", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137292025-04-30T06:43:32.286ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
137302025-04-30T06:43:32.302ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be911cbf-beaa-4e1f-ac4f-3331e957b04b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137312025-04-30T06:43:32.303ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
137322025-04-30T06:43:32.304ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d18f6b4-7a1a-4c0a-960e-ac200f222540", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137332025-04-30T06:43:32.304ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
137342025-04-30T06:43:32.305ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e210a38-0d0a-4cc6-9790-364bbdc4a116", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137352025-04-30T06:43:32.306ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
137362025-04-30T06:43:32.307ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce49f96f-ad91-40f9-9dd7-64b6efccd2f3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137372025-04-30T06:43:32.308ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
137382025-04-30T06:43:32.309ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dffc7224-8942-401c-b87d-76157ecda170", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137392025-04-30T06:43:32.309ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
137402025-04-30T06:43:32.311ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c46329a-7dd9-4ca8-9b83-9176826349f5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137412025-04-30T06:43:32.311ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
137422025-04-30T06:43:32.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2829fcc1-cdd6-4076-b05c-c19ce837efa6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137432025-04-30T06:43:32.313ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
137442025-04-30T06:43:32.317ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
137452025-04-30T06:43:32.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90ca59a8-caa6-40ea-9b13-017dfe2a4c41", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137462025-04-30T06:43:32.319ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
137472025-04-30T06:43:32.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e2d54b3-1860-4915-975e-b0a8566e0a5e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137482025-04-30T06:43:32.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
137492025-04-30T06:43:32.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba3cad6c-c420-479c-9dfc-ecc9270cacf3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137502025-04-30T06:43:32.323ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
137512025-04-30T06:43:32.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b5607d-ec99-41a5-99e3-8f977bd79d55", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137522025-04-30T06:43:32.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
137532025-04-30T06:43:32.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa7b92c-0224-46fd-ac31-36e00332cc88", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137542025-04-30T06:43:32.327ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
137552025-04-30T06:43:32.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df6c35b5-8c3b-492b-8f20-da9e1a359673", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137562025-04-30T06:43:32.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
137572025-04-30T06:43:32.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc4751b7-b798-41a0-859d-833b07e35c58", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137582025-04-30T06:43:32.329ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
137592025-04-30T06:43:32.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ff289f3-63ae-47fd-b994-94487b27b66b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137602025-04-30T06:43:32.330ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
137612025-04-30T06:43:32.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1982d355-bdf7-4e08-a730-9f8d91bc6a3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137622025-04-30T06:43:32.332ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
137632025-04-30T06:43:32.333ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59684cbd-9f73-4d72-b555-393acbe59ece", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137642025-04-30T06:43:32.333ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
137652025-04-30T06:43:32.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1715d13-253e-4fe2-8828-762e1dc63ac8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137662025-04-30T06:43:32.335ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
137672025-04-30T06:43:32.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fd80abf-820a-4f53-80fd-d94de74f289e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137682025-04-30T06:43:32.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
137692025-04-30T06:43:32.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9c54ff-12a5-4715-81eb-d00465ec8af1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137702025-04-30T06:43:32.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
137712025-04-30T06:43:32.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eef9eff1-efc3-4f2f-92e2-d09cccac8997", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137722025-04-30T06:43:32.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
137732025-04-30T06:43:32.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a451865-a880-4d89-9236-c871a2082184", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137742025-04-30T06:43:32.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
137752025-04-30T06:43:32.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21dc1db3-b8d2-4239-8426-0bd6916898d5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137762025-04-30T06:43:32.344ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
137772025-04-30T06:43:32.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d9c1e51-8782-4928-9bb5-01f65c24e75c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137782025-04-30T06:43:32.345ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
137792025-04-30T06:43:32.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ecc4b96-64d3-4af7-b3fe-6ae8c2fe9765", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137802025-04-30T06:43:32.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
137812025-04-30T06:43:32.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fff9b986-eef7-4413-993e-907e6c461687", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137822025-04-30T06:43:32.347ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
137832025-04-30T06:43:32.348ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0cc41f6-61de-4485-975d-f02d08d883b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
137842025-04-30T06:43:32.348ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137852025-04-30T06:43:32.348ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137862025-04-30T06:43:32.348ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137872025-04-30T06:43:32.348ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137882025-04-30T06:43:32.349ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137892025-04-30T06:43:32.351ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137902025-04-30T06:43:32.355ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137912025-04-30T06:43:32.355ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137922025-04-30T06:43:32.355ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137932025-04-30T06:43:32.355ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137942025-04-30T06:43:32.355ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137952025-04-30T06:43:32.357ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137962025-04-30T06:43:32.359ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137972025-04-30T06:43:32.362ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137982025-04-30T06:43:32.362ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137992025-04-30T06:43:32.363ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138002025-04-30T06:43:32.363ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138012025-04-30T06:43:32.441ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
138022025-04-30T06:43:32.441ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
138032025-04-30T06:43:32.442ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138042025-04-30T06:43:32.442ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138052025-04-30T06:43:32.442ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138062025-04-30T06:43:32.442ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138072025-04-30T06:43:32.443ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138082025-04-30T06:43:32.443ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:33403 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138092025-04-30T06:43:32.444ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:33403 req_id = 09e4c2fe-f868-4493-9807-1995a91c380c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138102025-04-30T06:43:32.444ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 601 local_addr = [::1]:45714 method = GET remote_addr = [::1]:33403 req_id = 09e4c2fe-f868-4493-9807-1995a91c380c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138112025-04-30T06:43:32.445ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e4c2fe-f868-4493-9807-1995a91c380c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138122025-04-30T06:43:32.447ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138132025-04-30T06:43:32.447ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138142025-04-30T06:43:32.447ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
138152025-04-30T06:43:32.449ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138162025-04-30T06:43:32.449ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138172025-04-30T06:43:32.449ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138182025-04-30T06:43:32.449ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138192025-04-30T06:43:32.450ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138202025-04-30T06:43:32.451ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138212025-04-30T06:43:32.471ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138222025-04-30T06:43:32.471ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138232025-04-30T06:43:32.471ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138242025-04-30T06:43:32.471ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138252025-04-30T06:43:32.473ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138262025-04-30T06:43:32.477ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138272025-04-30T06:43:32.477ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138282025-04-30T06:43:32.477ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138292025-04-30T06:43:32.477ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138302025-04-30T06:43:32.491ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138312025-04-30T06:43:32.491ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138322025-04-30T06:43:32.492ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138332025-04-30T06:43:32.520ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
138342025-04-30T06:43:32.521ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
138352025-04-30T06:43:32.521ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 515.628696ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
138362025-04-30T06:43:32.521ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
138372025-04-30T06:43:32.521ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138382025-04-30T06:43:32.521ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138392025-04-30T06:43:32.521ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138402025-04-30T06:43:32.521ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138412025-04-30T06:43:32.522ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138422025-04-30T06:43:32.523ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
138432025-04-30T06:43:32.525ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138442025-04-30T06:43:32.529ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138452025-04-30T06:43:32.529ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138462025-04-30T06:43:32.530ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138472025-04-30T06:43:32.530ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138482025-04-30T06:43:32.533ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
138492025-04-30T06:43:32.534ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
138502025-04-30T06:43:32.535ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 90 e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54058 req_id = 5ce39a25-d7bb-49d5-8fe2-1c422cfaf789
138512025-04-30T06:43:32.535ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37088,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54058 req_id = 5ce39a25-d7bb-49d5-8fe2-1c422cfaf789
138522025-04-30T06:43:32.535ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
138532025-04-30T06:43:32.536ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138542025-04-30T06:43:32.536ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54058 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5ce39a25-d7bb-49d5-8fe2-1c422cfaf789
138552025-04-30T06:43:32.536ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4277, tv_nsec: 335534617 } })
138562025-04-30T06:43:32.536ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
138572025-04-30T06:43:32.537ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fc 65 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .e..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53931 req_id = d5bda4bd-292a-47d0-8add-efcc784cb895
138582025-04-30T06:43:32.537ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64613,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53931 req_id = d5bda4bd-292a-47d0-8add-efcc784cb895
138592025-04-30T06:43:32.537ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138602025-04-30T06:43:32.537ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53931 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d5bda4bd-292a-47d0-8add-efcc784cb895
138612025-04-30T06:43:32.626ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
138622025-04-30T06:43:32.626ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138632025-04-30T06:43:32.627ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138642025-04-30T06:43:32.627ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138652025-04-30T06:43:32.627ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138662025-04-30T06:43:32.629ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138672025-04-30T06:43:32.630ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1c 8d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56143 req_id = b3510f78-60fb-496e-8634-603a01834172
138682025-04-30T06:43:32.630ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7309,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56143 req_id = b3510f78-60fb-496e-8634-603a01834172
138692025-04-30T06:43:32.631ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
138702025-04-30T06:43:32.631ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7309, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56143 req_id = b3510f78-60fb-496e-8634-603a01834172
138712025-04-30T06:43:32.631ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
138722025-04-30T06:43:32.632ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
138732025-04-30T06:43:32.632ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
138742025-04-30T06:43:32.638ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138752025-04-30T06:43:32.650ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138762025-04-30T06:43:32.650ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138772025-04-30T06:43:32.650ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138782025-04-30T06:43:32.651ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138792025-04-30T06:43:32.651ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0827d8f-e1b2-4a71-9153-1c6ba67ca34a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138802025-04-30T06:43:32.652ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
138812025-04-30T06:43:32.655ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138822025-04-30T06:43:32.656ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
138832025-04-30T06:43:32.656ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138842025-04-30T06:43:32.659ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8770f108-7447-4b57-867d-175674dc0ce5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138852025-04-30T06:43:32.659ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
138862025-04-30T06:43:32.663ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82c818ab-50fa-419c-8c50-463bf78c4c34", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138872025-04-30T06:43:32.664ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
138882025-04-30T06:43:32.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a5b9b5-73cd-415d-b318-9f1327bc82e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138892025-04-30T06:43:32.667ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
138902025-04-30T06:43:32.671ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52e91601-e670-4449-a88f-689f42c9d400", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138912025-04-30T06:43:32.672ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
138922025-04-30T06:43:32.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10705caf-3e28-41f1-8342-c4239eda9198", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:31 GMT"} })
138932025-04-30T06:43:32.675ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
138942025-04-30T06:43:32.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae43692b-c4e3-4a40-81ea-8a2ce273cb2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
138952025-04-30T06:43:32.682ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
138962025-04-30T06:43:32.683ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
138972025-04-30T06:43:32.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e41bc8-3660-4072-ac20-f7b017136168", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
138982025-04-30T06:43:32.685ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
138992025-04-30T06:43:32.686ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f1afea-bb5d-482b-8370-97699e8ac9db", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139002025-04-30T06:43:32.687ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
139012025-04-30T06:43:32.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "badf43aa-60aa-4c33-80ff-246094100021", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139022025-04-30T06:43:32.688ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
139032025-04-30T06:43:32.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0504a548-525c-4a06-a00b-e4e816dcc494", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139042025-04-30T06:43:32.689ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
139052025-04-30T06:43:32.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f94b4c00-8dc0-47d4-9c7b-8db671417548", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139062025-04-30T06:43:32.690ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
139072025-04-30T06:43:32.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77412d1-0eaf-4bac-b0f4-2436a9cea0a5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139082025-04-30T06:43:32.692ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
139092025-04-30T06:43:32.695ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "022f052d-e292-462c-90c9-f538ed2ed75b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139102025-04-30T06:43:32.695ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
139112025-04-30T06:43:32.697ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d34602f2-1d8d-478e-b47b-c9475567d133", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139122025-04-30T06:43:32.698ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
139132025-04-30T06:43:32.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a4b8430-164c-4962-8d34-d8b8a6b55e84", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139142025-04-30T06:43:32.699ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
139152025-04-30T06:43:32.703ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139162025-04-30T06:43:32.704ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139172025-04-30T06:43:32.706ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139182025-04-30T06:43:32.707ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139192025-04-30T06:43:32.707ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139202025-04-30T06:43:32.707ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139212025-04-30T06:43:32.708ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c41bd2-ef24-472b-a008-ddfda9497acd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139222025-04-30T06:43:32.709ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
139232025-04-30T06:43:32.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1768cb31-bf75-4bf2-b542-034be79f3aa6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139242025-04-30T06:43:32.716ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
139252025-04-30T06:43:32.718ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb8a591-2e48-4958-ae6c-78951c76cbd3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139262025-04-30T06:43:32.719ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
139272025-04-30T06:43:32.719ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1872172c-de47-46f8-9b82-dd71da8e038d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139282025-04-30T06:43:32.720ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
139292025-04-30T06:43:32.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9f208e-a753-42dd-a38f-43cc31e48b65", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139302025-04-30T06:43:32.721ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
139312025-04-30T06:43:32.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60460bdc-af5b-4b28-8346-67d77b982cb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139322025-04-30T06:43:32.723ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
139332025-04-30T06:43:32.724ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81c83985-a52f-4039-8124-e611e30ced0b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139342025-04-30T06:43:32.725ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
139352025-04-30T06:43:32.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a38f33f-1046-461b-8586-3df803d59c55", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139362025-04-30T06:43:32.726ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
139372025-04-30T06:43:32.729ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d933ba48-4d0d-411a-a74a-38f3f15a5bbe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139382025-04-30T06:43:32.730ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
139392025-04-30T06:43:32.732ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09b89c26-cd64-41e4-a9e2-92cc91f50a82", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139402025-04-30T06:43:32.733ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
139412025-04-30T06:43:32.733ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139422025-04-30T06:43:32.734ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139432025-04-30T06:43:32.734ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139442025-04-30T06:43:32.734ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
139452025-04-30T06:43:32.740ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139462025-04-30T06:43:32.742ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae1860c-a283-4f22-a528-a440d2b5e210", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139472025-04-30T06:43:32.742ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
139482025-04-30T06:43:32.743ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139492025-04-30T06:43:32.764ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139502025-04-30T06:43:32.764ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139512025-04-30T06:43:32.764ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
139522025-04-30T06:43:32.766ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139532025-04-30T06:43:32.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "105d8844-480a-4668-8e36-7c19b20db6ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139542025-04-30T06:43:32.769ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
139552025-04-30T06:43:32.770ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139562025-04-30T06:43:32.770ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
139572025-04-30T06:43:32.772ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139582025-04-30T06:43:32.812ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
139592025-04-30T06:43:32.813ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 81 fe 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59360 req_id = eeb995b8-8c17-4df6-a40a-ff9637c10c0e
139602025-04-30T06:43:32.813ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33278,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59360 req_id = eeb995b8-8c17-4df6-a40a-ff9637c10c0e
139612025-04-30T06:43:32.815ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
139622025-04-30T06:43:32.816ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139632025-04-30T06:43:32.816ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59360 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = eeb995b8-8c17-4df6-a40a-ff9637c10c0e
139642025-04-30T06:43:32.817ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4277, tv_nsec: 616322819 } })
139652025-04-30T06:43:32.818ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
139662025-04-30T06:43:32.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a93e589-80ec-4979-84ce-2639d03e181c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139672025-04-30T06:43:32.819ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
139682025-04-30T06:43:32.820ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64087 req_id = 15327bc5-8a64-4e24-a3fc-a136859e181d
139692025-04-30T06:43:32.820ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41717,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64087 req_id = 15327bc5-8a64-4e24-a3fc-a136859e181d
139702025-04-30T06:43:32.820ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139712025-04-30T06:43:32.821ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64087 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 15327bc5-8a64-4e24-a3fc-a136859e181d
139722025-04-30T06:43:32.822ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
139732025-04-30T06:43:32.881ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
139742025-04-30T06:43:32.881ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
139752025-04-30T06:43:32.882ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139762025-04-30T06:43:32.884ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139772025-04-30T06:43:32.885ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
139782025-04-30T06:43:32.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d816a6b2-b924-4011-a20f-e964b60bc6ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139792025-04-30T06:43:32.891ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
139802025-04-30T06:43:32.892ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
139812025-04-30T06:43:32.893ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ac dc 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58423 req_id = 9b500bfa-a741-440e-b7fe-50af01df3b19
139822025-04-30T06:43:32.894ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44252,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58423 req_id = 9b500bfa-a741-440e-b7fe-50af01df3b19
139832025-04-30T06:43:32.894ZDEBGtest_snapshot_unwind (store): query key key = _dendrite._tcp
139842025-04-30T06:43:32.894ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139852025-04-30T06:43:32.894ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58423 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9b500bfa-a741-440e-b7fe-50af01df3b19
139862025-04-30T06:43:32.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e33328d1-f5a3-49d8-ab12-2888f19f0e1c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
139872025-04-30T06:43:32.895ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
139882025-04-30T06:43:32.896ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48618, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4277, tv_nsec: 695208163 } })
139892025-04-30T06:43:32.896ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
139902025-04-30T06:43:32.897ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6e 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e nJ..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60472 req_id = ff91cb6e-641f-4b4e-83b1-9715dbaa52a1
139912025-04-30T06:43:32.897ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28234,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60472 req_id = ff91cb6e-641f-4b4e-83b1-9715dbaa52a1
139922025-04-30T06:43:32.897ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139932025-04-30T06:43:32.898ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60472 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ff91cb6e-641f-4b4e-83b1-9715dbaa52a1
139942025-04-30T06:43:32.898ZINFOtest_snapshot_unwind (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
139952025-04-30T06:43:32.948ZINFOtest_snapshot_unwind (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
139962025-04-30T06:43:32.948ZDEBGtest_snapshot_unwind (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
139972025-04-30T06:43:32.948ZDEBGtest_snapshot_unwind (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139982025-04-30T06:43:32.948ZWARNtest_snapshot_unwind (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139992025-04-30T06:43:32.950ZWARNtest_snapshot_unwind (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
140002025-04-30T06:43:32.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b65934e-c262-44a9-9a3c-ec0f387e065d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140012025-04-30T06:43:32.950ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
140022025-04-30T06:43:32.952ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64558ad6-a045-45b6-b361-baab32462648", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140032025-04-30T06:43:32.952ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
140042025-04-30T06:43:32.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed1513a-35d7-416a-9f02-e9d57e573fef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140052025-04-30T06:43:32.954ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
140062025-04-30T06:43:32.958ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
140072025-04-30T06:43:32.960ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84bf4bfa-5319-4c0c-b8d7-4a705193e5c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140082025-04-30T06:43:32.961ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
140092025-04-30T06:43:32.964ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e1aeff4-d9a2-4f79-b7ea-470527b2a923", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140102025-04-30T06:43:32.965ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
140112025-04-30T06:43:32.965ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:57433 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140122025-04-30T06:43:32.969ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:57433 req_id = a2346c87-f33c-4fc8-9314-27a5dd969c44 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140132025-04-30T06:43:32.969ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 556 local_addr = [::1]:61186 method = GET remote_addr = [::1]:57433 req_id = a2346c87-f33c-4fc8-9314-27a5dd969c44 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140142025-04-30T06:43:32.969ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:33443 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140152025-04-30T06:43:32.970ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:45878 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140162025-04-30T06:43:32.970ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:43510 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140172025-04-30T06:43:32.970ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:33443 req_id = b6d2bf64-eaba-4c8e-94d4-490a64044ed9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140182025-04-30T06:43:32.970ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 270 local_addr = [::1]:61186 method = GET remote_addr = [::1]:33443 req_id = b6d2bf64-eaba-4c8e-94d4-490a64044ed9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140192025-04-30T06:43:32.971ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:45878 req_id = 33c21d7d-e4dc-4d0b-baa0-cbafb57aa5e5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140202025-04-30T06:43:32.971ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:43510 req_id = 394e4c4a-0522-4f23-835b-7958e85208df sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140212025-04-30T06:43:32.971ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 489 local_addr = [::1]:61186 method = GET remote_addr = [::1]:45878 req_id = 33c21d7d-e4dc-4d0b-baa0-cbafb57aa5e5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140222025-04-30T06:43:32.971ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 429 local_addr = [::1]:61186 method = GET remote_addr = [::1]:43510 req_id = 394e4c4a-0522-4f23-835b-7958e85208df response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140232025-04-30T06:43:32.973ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:56514 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140242025-04-30T06:43:32.973ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:56514 req_id = f8440819-3414-488b-b256-c69dc0bf799b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140252025-04-30T06:43:32.973ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 134 local_addr = [::1]:61186 method = GET remote_addr = [::1]:56514 req_id = f8440819-3414-488b-b256-c69dc0bf799b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140262025-04-30T06:43:32.974ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:57787 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140272025-04-30T06:43:32.974ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:57787 req_id = f8acad1d-5415-4e07-8a90-6ed38ef37c76 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140282025-04-30T06:43:32.974ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 137 local_addr = [::1]:61186 method = GET remote_addr = [::1]:57787 req_id = f8acad1d-5415-4e07-8a90-6ed38ef37c76 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140292025-04-30T06:43:32.975ZDEBGtest_snapshot_unwind (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
140302025-04-30T06:43:32.975ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
140312025-04-30T06:43:32.976ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
140322025-04-30T06:43:32.978ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
140332025-04-30T06:43:32.979ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb2def31-40a6-4a65-9a89-78073ae11f8f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140342025-04-30T06:43:32.981ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
140352025-04-30T06:43:32.981ZINFOtest_snapshot_unwind (dropshot): accepted connection kind = pantry local_addr = [::1]:61186 remote_addr = [::1]:52010 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140362025-04-30T06:43:32.981ZTRACtest_snapshot_unwind (dropshot): incoming request kind = pantry local_addr = [::1]:61186 method = GET remote_addr = [::1]:52010 req_id = 85f6f64f-c57e-420b-adb3-c51367cdb941 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140372025-04-30T06:43:32.981ZINFOtest_snapshot_unwind (dropshot): request completed kind = pantry latency_us = 197 local_addr = [::1]:61186 method = GET remote_addr = [::1]:52010 req_id = 85f6f64f-c57e-420b-adb3-c51367cdb941 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
140382025-04-30T06:43:32.983ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 2 table_name = fields_string
140392025-04-30T06:43:32.983ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
140402025-04-30T06:43:32.987ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 1 table_name = measurements_i64
140412025-04-30T06:43:32.987ZTRACtest_snapshot_unwind (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
140422025-04-30T06:43:32.987ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b4b2ba4-0ec4-4d61-8daf-85033b2ed762", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140432025-04-30T06:43:32.987ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
140442025-04-30T06:43:32.990ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "621dfd74-9aae-4c7c-adc9-f313b8b6e820", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140452025-04-30T06:43:32.990ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
140462025-04-30T06:43:32.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28f70f13-7d74-488d-89a8-57381ba2cef1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140472025-04-30T06:43:32.992ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
140482025-04-30T06:43:32.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe5cc6f-4070-4657-bc61-0b752546ff9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140492025-04-30T06:43:32.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
140502025-04-30T06:43:32.996ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ec476a-35b8-4e2c-b912-693a6c53a00e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140512025-04-30T06:43:32.996ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
140522025-04-30T06:43:32.998ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "256beaab-eac4-414d-a2bb-014aa13c45b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140532025-04-30T06:43:32.999ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
140542025-04-30T06:43:33.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae03fc9-1d9e-4b9c-9ea6-438dcc5b7ac0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140552025-04-30T06:43:33.005ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
140562025-04-30T06:43:33.009ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79ff34a0-f2d2-4cf4-b80a-1cd4ddc925d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140572025-04-30T06:43:33.009ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140582025-04-30T06:43:33.009ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140592025-04-30T06:43:33.009ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140602025-04-30T06:43:33.009ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140612025-04-30T06:43:33.010ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140622025-04-30T06:43:33.011ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140632025-04-30T06:43:33.011ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140642025-04-30T06:43:33.011ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140652025-04-30T06:43:33.011ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
140662025-04-30T06:43:33.011ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140672025-04-30T06:43:33.014ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140682025-04-30T06:43:33.017ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140692025-04-30T06:43:33.017ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140702025-04-30T06:43:33.017ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140712025-04-30T06:43:33.017ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140722025-04-30T06:43:33.017ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140732025-04-30T06:43:33.017ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140742025-04-30T06:43:33.019ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140752025-04-30T06:43:33.025ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140762025-04-30T06:43:33.025ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140772025-04-30T06:43:33.025ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
140782025-04-30T06:43:33.026ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140792025-04-30T06:43:33.027ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140802025-04-30T06:43:33.030ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140812025-04-30T06:43:33.030ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140822025-04-30T06:43:33.030ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140832025-04-30T06:43:33.030ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140842025-04-30T06:43:33.033ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140852025-04-30T06:43:33.034ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
140862025-04-30T06:43:33.034ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140872025-04-30T06:43:33.086ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
140882025-04-30T06:43:33.086ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
140892025-04-30T06:43:33.087ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:56916 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140902025-04-30T06:43:33.088ZWARNtest_snapshot_unwind (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
140912025-04-30T06:43:33.088ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:56916 req_id = 7a8ca5c2-c65e-484d-9b29-13dcd866bf31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140922025-04-30T06:43:33.089ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 674 local_addr = [::1]:45714 method = GET remote_addr = [::1]:56916 req_id = 7a8ca5c2-c65e-484d-9b29-13dcd866bf31 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140932025-04-30T06:43:33.090ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a8ca5c2-c65e-484d-9b29-13dcd866bf31", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
140942025-04-30T06:43:33.092ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140952025-04-30T06:43:33.092ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140962025-04-30T06:43:33.092ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
140972025-04-30T06:43:33.093ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
140982025-04-30T06:43:33.093ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140992025-04-30T06:43:33.093ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141002025-04-30T06:43:33.093ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141012025-04-30T06:43:33.094ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141022025-04-30T06:43:33.100ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141032025-04-30T06:43:33.104ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141042025-04-30T06:43:33.104ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141052025-04-30T06:43:33.104ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141062025-04-30T06:43:33.109ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141072025-04-30T06:43:33.155ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
141082025-04-30T06:43:33.166ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
141092025-04-30T06:43:33.167ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
141102025-04-30T06:43:33.167ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 645.887037ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
141112025-04-30T06:43:33.167ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
141122025-04-30T06:43:33.167ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141132025-04-30T06:43:33.167ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141142025-04-30T06:43:33.167ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141152025-04-30T06:43:33.167ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141162025-04-30T06:43:33.168ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141172025-04-30T06:43:33.170ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141182025-04-30T06:43:33.176ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141192025-04-30T06:43:33.176ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141202025-04-30T06:43:33.176ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141212025-04-30T06:43:33.177ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141222025-04-30T06:43:33.185ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
141232025-04-30T06:43:33.185ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
141242025-04-30T06:43:33.185ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 50 17 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 P............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51225 req_id = 7570ac65-b5a5-4056-951f-c817fd85a779
141252025-04-30T06:43:33.185ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20503,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51225 req_id = 7570ac65-b5a5-4056-951f-c817fd85a779
141262025-04-30T06:43:33.186ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
141272025-04-30T06:43:33.187ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141282025-04-30T06:43:33.187ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51225 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7570ac65-b5a5-4056-951f-c817fd85a779
141292025-04-30T06:43:33.187ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4277, tv_nsec: 986635222 } })
141302025-04-30T06:43:33.188ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
141312025-04-30T06:43:33.188ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6b 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e kt..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58851 req_id = c05d3d1f-ba44-441d-b6a1-bc744e6543f8
141322025-04-30T06:43:33.188ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27508,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58851 req_id = c05d3d1f-ba44-441d-b6a1-bc744e6543f8
141332025-04-30T06:43:33.188ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141342025-04-30T06:43:33.188ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58851 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c05d3d1f-ba44-441d-b6a1-bc744e6543f8
141352025-04-30T06:43:33.268ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
141362025-04-30T06:43:33.268ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141372025-04-30T06:43:33.268ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141382025-04-30T06:43:33.269ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141392025-04-30T06:43:33.269ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
141402025-04-30T06:43:33.269ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141412025-04-30T06:43:33.270ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9d 19 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65154 req_id = a549bb5e-0ec6-4f6d-9b1e-391be76179f9
141422025-04-30T06:43:33.271ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40217,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65154 req_id = a549bb5e-0ec6-4f6d-9b1e-391be76179f9
141432025-04-30T06:43:33.271ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
141442025-04-30T06:43:33.271ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40217, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65154 req_id = a549bb5e-0ec6-4f6d-9b1e-391be76179f9
141452025-04-30T06:43:33.272ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
141462025-04-30T06:43:33.272ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
141472025-04-30T06:43:33.272ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
141482025-04-30T06:43:33.275ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141492025-04-30T06:43:33.281ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141502025-04-30T06:43:33.281ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141512025-04-30T06:43:33.281ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
141522025-04-30T06:43:33.281ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141532025-04-30T06:43:33.293ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141542025-04-30T06:43:33.293ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
141552025-04-30T06:43:33.294ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141562025-04-30T06:43:33.294ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "524b5ea6-1d26-4bfc-a84e-6d9118a76b73", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141572025-04-30T06:43:33.295ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
141582025-04-30T06:43:33.304ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17758945-938d-4c50-aa13-e9a06bf0b1e2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141592025-04-30T06:43:33.305ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
141602025-04-30T06:43:33.316ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0af7dbc5-5106-4108-85a4-cac1592d82f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141612025-04-30T06:43:33.317ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
141622025-04-30T06:43:33.321ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e34ce5fc-29d2-497a-8886-88fb139538b4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141632025-04-30T06:43:33.322ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
141642025-04-30T06:43:33.324ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc8f9611-9854-44f7-8924-dcf39211cc63", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141652025-04-30T06:43:33.325ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
141662025-04-30T06:43:33.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fa35b5d-aae5-401a-931a-e1480b5a3ce9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141672025-04-30T06:43:33.328ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
141682025-04-30T06:43:33.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93187a17-cdfe-4e3c-a98c-03593b518a1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141692025-04-30T06:43:33.331ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
141702025-04-30T06:43:33.334ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
141712025-04-30T06:43:33.334ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "872e23d7-2b38-43d7-af2b-d15e6a443e67", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141722025-04-30T06:43:33.334ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
141732025-04-30T06:43:33.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fd4ee28-6e05-463f-bb42-0282a837d756", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141742025-04-30T06:43:33.336ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
141752025-04-30T06:43:33.337ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb096dd2-74e2-4a90-89bf-675f74fbb2f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141762025-04-30T06:43:33.338ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
141772025-04-30T06:43:33.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c5cda0b-d55d-4bc3-ab1a-766b557a7e3a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141782025-04-30T06:43:33.339ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
141792025-04-30T06:43:33.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bc5958-80a4-4736-aceb-2b6877ab7edd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141802025-04-30T06:43:33.340ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
141812025-04-30T06:43:33.342ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b1b6ce2-0b6c-4389-b56a-ef2b2cb3eea4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141822025-04-30T06:43:33.343ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
141832025-04-30T06:43:33.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "708999f7-4ac7-46a8-904f-9febed604556", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141842025-04-30T06:43:33.346ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
141852025-04-30T06:43:33.348ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ffca87d-c5b7-42f6-835a-48007aa767e4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141862025-04-30T06:43:33.348ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
141872025-04-30T06:43:33.350ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eb319cd-9214-46da-95b5-232f112f8e3b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141882025-04-30T06:43:33.351ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
141892025-04-30T06:43:33.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da8cab92-f97f-4d0c-ac4a-732da97b36df", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141902025-04-30T06:43:33.353ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
141912025-04-30T06:43:33.356ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94174f6f-ab8d-434c-be2d-718897f718b7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141922025-04-30T06:43:33.356ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
141932025-04-30T06:43:33.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc02afd4-87c4-4d1e-b993-150f71d30d52", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141942025-04-30T06:43:33.359ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
141952025-04-30T06:43:33.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f60f0dc-dc1a-45a5-91ca-e3e2c31425ce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141962025-04-30T06:43:33.360ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
141972025-04-30T06:43:33.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fa6581f-b381-4cd6-b51f-e3d73687b6d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
141982025-04-30T06:43:33.362ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
141992025-04-30T06:43:33.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd3fb91-24c1-4c2e-8ac5-7e143b922ca6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142002025-04-30T06:43:33.363ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
142012025-04-30T06:43:33.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "430429c0-2d50-4e7e-8531-036c714ba3ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142022025-04-30T06:43:33.364ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
142032025-04-30T06:43:33.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd57184-8516-4354-a934-21a87a0a9152", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142042025-04-30T06:43:33.366ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
142052025-04-30T06:43:33.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8593b733-8f48-4a6b-a3b7-02454971f900", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142062025-04-30T06:43:33.369ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
142072025-04-30T06:43:33.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa979bc2-4cb8-44d7-b87c-2194909d31c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142082025-04-30T06:43:33.373ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
142092025-04-30T06:43:33.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd0b6010-9c17-466b-9e25-63951ed3ca13", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142102025-04-30T06:43:33.376ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
142112025-04-30T06:43:33.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0cb89ea-1c50-4e02-8278-7190b753c039", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142122025-04-30T06:43:33.378ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
142132025-04-30T06:43:33.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c7b11c4-5989-4698-92e5-659133e99ea4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142142025-04-30T06:43:33.381ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
142152025-04-30T06:43:33.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07727054-8630-4541-b453-4ca66efc52ce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142162025-04-30T06:43:33.384ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
142172025-04-30T06:43:33.385ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142182025-04-30T06:43:33.385ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142192025-04-30T06:43:33.385ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142202025-04-30T06:43:33.385ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
142212025-04-30T06:43:33.386ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142222025-04-30T06:43:33.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c113e7c-5a48-4f59-b664-1ac6216b3ff5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142232025-04-30T06:43:33.387ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
142242025-04-30T06:43:33.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c155f4-9ac9-4a76-8224-a92b42177476", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142252025-04-30T06:43:33.388ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
142262025-04-30T06:43:33.388ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142272025-04-30T06:43:33.393ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142282025-04-30T06:43:33.393ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142292025-04-30T06:43:33.394ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
142302025-04-30T06:43:33.394ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142312025-04-30T06:43:33.404ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e34280-05b1-4046-8996-2d78758a3850", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142322025-04-30T06:43:33.406ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
142332025-04-30T06:43:33.410ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142342025-04-30T06:43:33.410ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
142352025-04-30T06:43:33.411ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142362025-04-30T06:43:33.417ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c5641ec-0a40-4442-a8bc-3bc9567f35ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142372025-04-30T06:43:33.417ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
142382025-04-30T06:43:33.420ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5116682-e964-41f3-8b00-3d17ece5c210", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142392025-04-30T06:43:33.420ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
142402025-04-30T06:43:33.424ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "375d4e8f-c065-4ba6-addc-83baba3e5274", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142412025-04-30T06:43:33.424ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
142422025-04-30T06:43:33.427ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e865b57-0826-4df1-b1cb-fc640f4e634f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142432025-04-30T06:43:33.427ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
142442025-04-30T06:43:33.433ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bced9d02-27c9-44b3-a7dd-6c8ed43304e5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142452025-04-30T06:43:33.433ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
142462025-04-30T06:43:33.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61525e3e-c524-45b7-bea9-bd207933bc83", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142472025-04-30T06:43:33.437ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
142482025-04-30T06:43:33.442ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d537f6-a453-4d15-8670-1c541fbfed2b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142492025-04-30T06:43:33.443ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
142502025-04-30T06:43:33.447ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
142512025-04-30T06:43:33.451ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4e8f417-22ba-498f-a6ac-82c26f68d4e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142522025-04-30T06:43:33.451ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
142532025-04-30T06:43:33.453ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff078676-196d-4e22-9758-61e40a8587a6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142542025-04-30T06:43:33.453ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
142552025-04-30T06:43:33.454ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec6b714d-fd2f-431b-9cdb-e5a4268e7a5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142562025-04-30T06:43:33.454ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
142572025-04-30T06:43:33.456ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbedab56-b228-45d9-8044-f94c8188b25b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142582025-04-30T06:43:33.458ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
142592025-04-30T06:43:33.461ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a9c0e0-ff90-4b51-ae99-2c59814c0079", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142602025-04-30T06:43:33.462ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
142612025-04-30T06:43:33.462ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142622025-04-30T06:43:33.462ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
142632025-04-30T06:43:33.462ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
142642025-04-30T06:43:33.463ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
142652025-04-30T06:43:33.470ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142662025-04-30T06:43:33.475ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
142672025-04-30T06:43:33.475ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
142682025-04-30T06:43:33.475ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142692025-04-30T06:43:33.475ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
142702025-04-30T06:43:33.475ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
142712025-04-30T06:43:33.476ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
142722025-04-30T06:43:33.480ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142732025-04-30T06:43:33.486ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
142742025-04-30T06:43:33.486ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
142752025-04-30T06:43:33.487ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
142762025-04-30T06:43:33.487ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
142772025-04-30T06:43:33.547ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
142782025-04-30T06:43:33.547ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory
142792025-04-30T06:43:33.547ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142802025-04-30T06:43:33.547ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142812025-04-30T06:43:33.547ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142822025-04-30T06:43:33.547ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
142832025-04-30T06:43:33.549ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142842025-04-30T06:43:33.549ZINFOtest_snapshot_unwind (dropshot): accepted connection local_addr = [::1]:45714 remote_addr = [::1]:64463 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142852025-04-30T06:43:33.551ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = [::1]:45714 method = GET remote_addr = [::1]:64463 req_id = a8c8b2c3-2015-4c5d-a1f5-c77b8dd7cc0e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
142862025-04-30T06:43:33.552ZINFOtest_snapshot_unwind (dropshot): request completed latency_us = 500 local_addr = [::1]:45714 method = GET remote_addr = [::1]:64463 req_id = a8c8b2c3-2015-4c5d-a1f5-c77b8dd7cc0e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
142872025-04-30T06:43:33.552ZDEBGtest_snapshot_unwind (BackgroundTasks): client response SledAgent = http://[::1]:45714 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45714/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8c8b2c3-2015-4c5d-a1f5-c77b8dd7cc0e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
142882025-04-30T06:43:33.555ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
142892025-04-30T06:43:33.555ZDEBGtest_snapshot_unwind (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
142902025-04-30T06:43:33.555ZDEBGtest_snapshot_unwind (BackgroundTasks): finished collection background_task = inventory_collection
142912025-04-30T06:43:33.559ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
142922025-04-30T06:43:33.559ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142932025-04-30T06:43:33.559ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
142942025-04-30T06:43:33.559ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
142952025-04-30T06:43:33.560ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
142962025-04-30T06:43:33.561ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142972025-04-30T06:43:33.569ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142982025-04-30T06:43:33.569ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142992025-04-30T06:43:33.569ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
143002025-04-30T06:43:33.570ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143012025-04-30T06:43:33.572ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143022025-04-30T06:43:33.576ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143032025-04-30T06:43:33.576ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143042025-04-30T06:43:33.577ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143052025-04-30T06:43:33.577ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143062025-04-30T06:43:33.577ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143072025-04-30T06:43:33.577ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
143082025-04-30T06:43:33.580ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143092025-04-30T06:43:33.634ZINFOtest_snapshot_unwind (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
143102025-04-30T06:43:33.635ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143112025-04-30T06:43:33.636ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 11bd0ffa-09aa-4848-adec-c08f5d3b3161 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143122025-04-30T06:43:33.636ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 11bd0ffa-09aa-4848-adec-c08f5d3b3161 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143132025-04-30T06:43:33.636ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 11bd0ffa-09aa-4848-adec-c08f5d3b3161 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143142025-04-30T06:43:33.638ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 11bd0ffa-09aa-4848-adec-c08f5d3b3161 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143152025-04-30T06:43:33.671ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 35131 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 11bd0ffa-09aa-4848-adec-c08f5d3b3161 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
143162025-04-30T06:43:33.671ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "11bd0ffa-09aa-4848-adec-c08f5d3b3161", "content-length": "405", "date": "Wed, 30 Apr 2025 06:43:32 GMT"} })
143172025-04-30T06:43:33.672ZDEBGtest_snapshot_unwind (oximeter): client request body = None method = GET uri = http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143182025-04-30T06:43:33.675ZTRACtest_snapshot_unwind (dropshot_internal): incoming request local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 306df68e-229c-4436-b1df-3f80a516148f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143192025-04-30T06:43:33.676ZTRACtest_snapshot_unwind (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 306df68e-229c-4436-b1df-3f80a516148f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143202025-04-30T06:43:33.676ZDEBGtest_snapshot_unwind (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 306df68e-229c-4436-b1df-3f80a516148f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143212025-04-30T06:43:33.677ZDEBGtest_snapshot_unwind (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 306df68e-229c-4436-b1df-3f80a516148f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143222025-04-30T06:43:33.677ZWARNtest_snapshot_unwind (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
143232025-04-30T06:43:33.678ZWARNtest_snapshot_unwind (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
143242025-04-30T06:43:33.678ZDEBGtest_snapshot_unwind (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 510.978031ms iteration = 52 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
143252025-04-30T06:43:33.678ZDEBGtest_snapshot_unwind (BackgroundTasks): activating background_task = inventory_collection iteration = 53 reason = Signaled
143262025-04-30T06:43:33.678ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143272025-04-30T06:43:33.678ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143282025-04-30T06:43:33.678ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143292025-04-30T06:43:33.678ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143302025-04-30T06:43:33.679ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143312025-04-30T06:43:33.685ZINFOtest_snapshot_unwind (dropshot_internal): request completed latency_us = 9886 local_addr = 127.0.0.1:53046 method = GET remote_addr = 127.0.0.1:52439 req_id = 306df68e-229c-4436-b1df-3f80a516148f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D
143322025-04-30T06:43:33.689ZDEBGtest_snapshot_unwind (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:53046/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmN2E1NmNjNC02ODU1LTRlZTEtYmYyMy00M2M1ZjQzNzkxNWMifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "306df68e-229c-4436-b1df-3f80a516148f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143332025-04-30T06:43:33.692ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143342025-04-30T06:43:33.692ZTRACtest_snapshot_unwind (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143352025-04-30T06:43:33.692ZINFOtest_snapshot_unwind (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
143362025-04-30T06:43:33.692ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143372025-04-30T06:43:33.702ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143382025-04-30T06:43:33.705ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143392025-04-30T06:43:33.705ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143402025-04-30T06:43:33.706ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143412025-04-30T06:43:33.707ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143422025-04-30T06:43:33.707ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143432025-04-30T06:43:33.710ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143442025-04-30T06:43:33.711ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143452025-04-30T06:43:33.711ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143462025-04-30T06:43:33.711ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143472025-04-30T06:43:33.712ZDEBGtest_snapshot_unwind (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(7cce6308-ea2f-4bdc-bd6e-8cdb01380105, 0)]
143482025-04-30T06:43:33.712ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
143492025-04-30T06:43:33.713ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: da d0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55848 req_id = dd7d00ac-12f0-4376-9f13-cf5b0d444aad
143502025-04-30T06:43:33.714ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56016,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55848 req_id = dd7d00ac-12f0-4376-9f13-cf5b0d444aad
143512025-04-30T06:43:33.714ZDEBGtest_snapshot_unwind (store): query key key = _mgs._tcp
143522025-04-30T06:43:33.714ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143532025-04-30T06:43:33.715ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55848 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dd7d00ac-12f0-4376-9f13-cf5b0d444aad
143542025-04-30T06:43:33.717ZDEBGtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64867, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4278, tv_nsec: 516084436 } })
143552025-04-30T06:43:33.717ZTRACtest_snapshot_unwind (DnsResolver): lookup_service_targets: looking up SRV target
143562025-04-30T06:43:33.717ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8a d3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56109 req_id = c502d83f-2248-46d7-ba9a-8d9761fd1c20
143572025-04-30T06:43:33.717ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35539,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56109 req_id = c502d83f-2248-46d7-ba9a-8d9761fd1c20
143582025-04-30T06:43:33.718ZDEBGtest_snapshot_unwind (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143592025-04-30T06:43:33.718ZDEBGtest_snapshot_unwind (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56109 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c502d83f-2248-46d7-ba9a-8d9761fd1c20
143602025-04-30T06:43:33.719ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
143612025-04-30T06:43:33.785ZTRACtest_snapshot_unwind (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
143622025-04-30T06:43:33.786ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143632025-04-30T06:43:33.787ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143642025-04-30T06:43:33.787ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143652025-04-30T06:43:33.787ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
143662025-04-30T06:43:33.789ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143672025-04-30T06:43:33.789ZDEBGtest_snapshot_unwind (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143682025-04-30T06:43:33.789ZDEBGtest_snapshot_unwind (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143692025-04-30T06:43:33.790ZTRACtest_snapshot_unwind (dropshot): incoming request local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = e04adce1-794a-4809-a437-607156a95001 uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
143702025-04-30T06:43:33.791ZTRACtest_snapshot_unwind (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 38 35 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 85..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59360 req_id = 9170ca4b-dd0b-4c66-91da-abacfd4627b5
143712025-04-30T06:43:33.791ZDEBGtest_snapshot_unwind (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14389,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59360 req_id = 9170ca4b-dd0b-4c66-91da-abacfd4627b5
143722025-04-30T06:43:33.791ZDEBGtest_snapshot_unwind (store): query key key = _clickhouse-admin-keeper._tcp
143732025-04-30T06:43:33.791ZERROtest_snapshot_unwind (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14389, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59360 req_id = 9170ca4b-dd0b-4c66-91da-abacfd4627b5
143742025-04-30T06:43:33.797ZINFOtest_snapshot_unwind (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6539 local_addr = 127.0.0.1:49133 method = GET remote_addr = 127.0.0.1:60571 req_id = e04adce1-794a-4809-a437-607156a95001 response_code = 200 uri = /f7a56cc4-6855-4ee1-bf23-43c5f437915c
143752025-04-30T06:43:33.800ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection background_task = inventory_collection
143762025-04-30T06:43:33.800ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:64867
143772025-04-30T06:43:33.800ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/ignition
143782025-04-30T06:43:33.801ZTRACtest_snapshot_unwind (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143792025-04-30T06:43:33.810ZDEBGtest_snapshot_unwind (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143802025-04-30T06:43:33.810ZTRACtest_snapshot_unwind (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143812025-04-30T06:43:33.810ZTRACtest_snapshot_unwind (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = f7a56cc4-6855-4ee1-bf23-43c5f437915c
143822025-04-30T06:43:33.810ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143832025-04-30T06:43:33.817ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143842025-04-30T06:43:33.817ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143852025-04-30T06:43:33.817ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
143862025-04-30T06:43:33.818ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143872025-04-30T06:43:33.822ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143882025-04-30T06:43:33.822ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
143892025-04-30T06:43:33.824ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143902025-04-30T06:43:33.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0750979b-5e03-4e7f-8512-e717c4f1e1ab", "content-length": "835", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143912025-04-30T06:43:33.824ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0
143922025-04-30T06:43:33.827ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8041ccc1-1282-450f-9bfa-eb4429ba9c2a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143932025-04-30T06:43:33.827ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0
143942025-04-30T06:43:33.829ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db10df2-c574-4312-939b-33b05c9e2336", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143952025-04-30T06:43:33.830ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1
143962025-04-30T06:43:33.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa7ce51-910c-4cd0-9bf9-c69f9ba7d8a0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143972025-04-30T06:43:33.834ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0
143982025-04-30T06:43:33.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d676fc0e-ac32-4a11-a6fa-af2f6d7a3bd0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
143992025-04-30T06:43:33.840ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1
144002025-04-30T06:43:33.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5241cb6c-22b4-4a7b-84b6-d9d31b2f12f5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144012025-04-30T06:43:33.843ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0
144022025-04-30T06:43:33.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41607afb-9452-41c5-91dc-546a4379f866", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144032025-04-30T06:43:33.849ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1
144042025-04-30T06:43:33.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f20a1e04-bbbd-4458-9919-1052406977a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144052025-04-30T06:43:33.852ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cmpa
144062025-04-30T06:43:33.853ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06ab9eff-1f99-474b-8e80-68901dd0daaf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144072025-04-30T06:43:33.853ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
144082025-04-30T06:43:33.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5192e8c-d28d-4f81-8934-8b544a1c6ff2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144092025-04-30T06:43:33.857ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
144102025-04-30T06:43:33.858ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
144112025-04-30T06:43:33.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1cfb2d1-91b5-42ef-9857-5a23a803987c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144122025-04-30T06:43:33.861ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/0/component/rot/cfpa
144132025-04-30T06:43:33.862ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f8f9e2-8bb4-4338-b790-ea686a7cfbd7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144142025-04-30T06:43:33.862ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1
144152025-04-30T06:43:33.864ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32bb5f9b-539a-444f-a106-d6cc00e6d00f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144162025-04-30T06:43:33.864ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0
144172025-04-30T06:43:33.866ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f364a5c-63e5-4557-a352-6891f8992a81", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144182025-04-30T06:43:33.866ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1
144192025-04-30T06:43:33.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de7b88a8-0cd4-47be-a741-5ee62efc57a0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144202025-04-30T06:43:33.868ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0
144212025-04-30T06:43:33.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6e79dbf-e78d-4fa3-99f9-c94d37e53060", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144222025-04-30T06:43:33.870ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1
144232025-04-30T06:43:33.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8827ce1-6282-447a-9619-ffe1b3215de3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144242025-04-30T06:43:33.872ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0
144252025-04-30T06:43:33.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c71e810b-4ea8-41d6-ab16-f2790c8f9735", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144262025-04-30T06:43:33.874ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1
144272025-04-30T06:43:33.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1046c6c4-2cfd-4a91-9c22-2862faff4f2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144282025-04-30T06:43:33.876ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cmpa
144292025-04-30T06:43:33.877ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd01c07-02ec-4cd2-a093-a8c6dcd6a984", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144302025-04-30T06:43:33.878ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
144312025-04-30T06:43:33.879ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fab19bc4-0523-4a88-9e8c-e2e7ee07ea75", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144322025-04-30T06:43:33.880ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
144332025-04-30T06:43:33.881ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e45935c8-16bd-4332-b881-f73e987b1238", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144342025-04-30T06:43:33.881ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/switch/1/component/rot/cfpa
144352025-04-30T06:43:33.886ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7ab02d-5c73-49e3-a7e4-4fae5d8f4ea3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144362025-04-30T06:43:33.886ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0
144372025-04-30T06:43:33.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7945726-a1c9-44f3-9221-6805010f3a31", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144382025-04-30T06:43:33.888ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0
144392025-04-30T06:43:33.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f753d4e-efbe-455b-8bf9-df41c82c2059", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144402025-04-30T06:43:33.890ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1
144412025-04-30T06:43:33.893ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cbb1a4-52cb-4d1c-a700-2d7cb70aa35f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144422025-04-30T06:43:33.893ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0
144432025-04-30T06:43:33.896ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d3099d-9359-40fb-aa27-17cda4a9ee67", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144442025-04-30T06:43:33.896ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1
144452025-04-30T06:43:33.899ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "928fe3d2-e6d5-4a4c-9620-5009d4085646", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144462025-04-30T06:43:33.899ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0
144472025-04-30T06:43:33.901ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6953b4e6-8de7-4d8e-adc0-df2b144a88ce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144482025-04-30T06:43:33.901ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1
144492025-04-30T06:43:33.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb38c114-4805-42ef-bd46-d98b4a6304f9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144502025-04-30T06:43:33.903ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cmpa
144512025-04-30T06:43:33.904ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "415e3d7b-e6fe-4516-85bc-89adac951d3d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144522025-04-30T06:43:33.904ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
144532025-04-30T06:43:33.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02951238-43bf-4146-a8fa-b66f60d59c2a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144542025-04-30T06:43:33.906ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
144552025-04-30T06:43:33.907ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74e009f8-d7cd-4afd-b01b-9a2a5e7140fa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144562025-04-30T06:43:33.907ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/0/component/rot/cfpa
144572025-04-30T06:43:33.908ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
144582025-04-30T06:43:33.909ZTRACtest_snapshot_unwind (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144592025-04-30T06:43:33.909ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144602025-04-30T06:43:33.909ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
144612025-04-30T06:43:33.911ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144622025-04-30T06:43:33.914ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03786a14-d150-4c3f-ae65-0791ae4b8bac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144632025-04-30T06:43:33.915ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1
144642025-04-30T06:43:33.917ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144652025-04-30T06:43:33.926ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144662025-04-30T06:43:33.927ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144672025-04-30T06:43:33.927ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
144682025-04-30T06:43:33.928ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144692025-04-30T06:43:33.928ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa1dabf-9369-4ace-8872-bdcb1a12149d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144702025-04-30T06:43:33.929ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0
144712025-04-30T06:43:33.934ZTRACtest_snapshot_unwind (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144722025-04-30T06:43:33.934ZDEBGtest_snapshot_unwind (InternalApi): roles roles = RoleSet { roles: {} }
144732025-04-30T06:43:33.935ZDEBGtest_snapshot_unwind (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144742025-04-30T06:43:33.938ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "142dda85-7bd2-41c4-9060-168717854954", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144752025-04-30T06:43:33.938ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1
144762025-04-30T06:43:33.944ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6535f5b9-e06d-49a6-a172-6043bdaf8efc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144772025-04-30T06:43:33.946ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0
144782025-04-30T06:43:33.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c846a18c-0b51-4856-9e3a-46a9e4e621eb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144792025-04-30T06:43:33.951ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1
144802025-04-30T06:43:33.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c2a669c-0c93-4eac-927e-89418150f526", "content-length": "169", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144812025-04-30T06:43:33.955ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0
144822025-04-30T06:43:33.966ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95295a89-ee0c-4890-ae1a-9fadfef747e0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144832025-04-30T06:43:33.967ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1
144842025-04-30T06:43:33.974ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "751aa699-8878-4c9c-adbe-347b191ec442", "content-length": "178", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144852025-04-30T06:43:33.974ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cmpa
144862025-04-30T06:43:33.977ZDEBGtest_snapshot_unwind (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
144872025-04-30T06:43:33.977ZDEBGtest_snapshot_unwind (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
144882025-04-30T06:43:33.978ZDEBGtest_snapshot_unwind (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38
144892025-04-30T06:43:33.991ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
144902025-04-30T06:43:33.993ZDEBGtest_snapshot_unwind: attempt to fetch latest inventory collection result = Ok("found")
144912025-04-30T06:43:33.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4abf69-a44b-4990-a103-8c75dfac33af", "content-length": "702", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144922025-04-30T06:43:33.995ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
144932025-04-30T06:43:33.997ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 12 table_name = fields_string
144942025-04-30T06:43:33.997ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
144952025-04-30T06:43:33.997ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2408c493-53b5-4702-a734-195eaabbffe0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144962025-04-30T06:43:33.998ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
144972025-04-30T06:43:33.999ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b9505ca-9ad5-4f89-95cd-9cd19ea127ec", "content-length": "729", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
144982025-04-30T06:43:33.999ZDEBGtest_snapshot_unwind (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:64867 method = GET uri = http://[::1]:64867/sp/sled/1/component/rot/cfpa
144992025-04-30T06:43:34.001ZDEBGtest_snapshot_unwind (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:64867 result = Ok(Response { url: "http://[::1]:64867/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a68199-1cab-46dc-9303-cd3cf80a0933", "content-length": "728", "date": "Wed, 30 Apr 2025 06:43:33 GMT"} })
145002025-04-30T06:43:34.001ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145012025-04-30T06:43:34.001ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145022025-04-30T06:43:34.001ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145032025-04-30T06:43:34.001ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145042025-04-30T06:43:34.002ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145052025-04-30T06:43:34.003ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = fields_u16
145062025-04-30T06:43:34.003ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
145072025-04-30T06:43:34.005ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145082025-04-30T06:43:34.008ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145092025-04-30T06:43:34.008ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145102025-04-30T06:43:34.008ZTRACtest_snapshot_unwind (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145112025-04-30T06:43:34.008ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145122025-04-30T06:43:34.008ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145132025-04-30T06:43:34.009ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145142025-04-30T06:43:34.010ZDEBGtest_snapshot_unwind (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_rows = 5 table_name = fields_uuid
145152025-04-30T06:43:34.010ZTRACtest_snapshot_unwind (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 1fb568f8-f143-48f3-a3bc-42a14be34a38 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
145162025-04-30T06:43:34.012ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145172025-04-30T06:43:34.015ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145182025-04-30T06:43:34.016ZTRACtest_snapshot_unwind (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145192025-04-30T06:43:34.016ZDEBGtest_snapshot_unwind (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145202025-04-30T06:43:34.016ZDEBGtest_snapshot_unwind (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145212025-04-30T06:43:34.099ZDEBGtest_snapshot_unwind (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45714
145222025-04-30T06:43:34.099ZDEBGtest_snapshot_unwind (BackgroundTasks): client request SledAgent = http://[::1]:45714 background_task = inventory_collection body = None method = GET uri = http://[::1]:45714/inventory